Why can't you just run Q-learning on a fixed dataset?
hardAnswer
- Because the maximization step queries actions the dataset never contains.
- The target takes a maximum over actions, and for unseen actions the network's value is an extrapolation with no data to correct it, so overestimation errors get selected precisely because they are overestimates.
- Bootstrapping then propagates those inflated values through the dataset, and the learned policy prefers exactly the actions nobody ever tried.
- Without environment interaction there is no corrective feedback loop, which is what makes offline learning qualitatively different from off-policy learning with a replay buffer.
- The fixes constrain the policy to the data: behaviour regularization, conservative value penalties on out-of-distribution actions, or filtered imitation of the best observed trajectories.
Check yourself — multiple choice
- It works fine
- The max operator queries unseen actions, whose values are unconstrained extrapolations, and overestimates are preferentially selected and bootstrapped with no interaction to correct them
- The dataset is always too small
- Only the discount factor needs adjusting
Distributional shift over actions plus no corrective feedback makes naive offline Q-learning diverge towards untried actions.
#offline-rl#theory
Practise Reinforcement Learning
214 interview questions in this topic.