Why is distribution shift the core challenge in offline RL?
hardAnswer
- Policy π_θ may query Q at (s, a) never seen in data → Q's extrapolation is nonsense → policy exploits Q's overestimation of OOD actions → learned policy fails.
- Naive off-policy methods (SAC on offline data) diverge catastrophically.
- Fixes: constrain policy near behavior (BCQ), pessimistic Q (CQL), model-based with uncertainty (MOReL, COMBO), decision-transformer (no Q at all).
Check yourself — multiple choice
- Random
- Policy queries Q at unseen OOD (s, a) → nonsense extrapolation → policy exploits overestimation → catastrophic divergence; fix via policy constraints (BCQ) / pessimistic Q (CQL) / MBRL uncertainty
- Same as online
- Not real
Offline RL: OOD action extrapolation → catastrophic exploitation; constrain or pessimism.
#offline-rl
Practise Reinforcement Learning
214 interview questions in this topic.