What is off-policy evaluation and why is it hard?
hardAnswer
- Off-policy evaluation estimates the value of a target policy using data collected by a different behavior policy — without deploying the new policy.
- It's crucial in medicine, finance, and any high-stakes system where trying the new policy is risky.
- It is hard because of distribution shift; techniques include importance sampling, doubly robust estimators, and model-based methods.
- Variance and bias tradeoffs are severe.
Check yourself — multiple choice
- OPE requires deploying the new policy to collect data
- Importance sampling reweights returns to correct for policy mismatch
- OPE is easy in high-dimensional problems
- Doubly robust estimators use only importance sampling
Importance sampling corrects for the distribution difference between policies.
#evaluation#safety
Practise Reinforcement Learning
214 interview questions in this topic.