Why is naive DQN unstable and what specific fixes address which cause?
hardAnswer
- (1) Correlated sequential data → replay buffer breaks correlation.
- (2) Moving target Q'-target = Q → target network freezes target for N steps.
- (3) Overestimation from noise in max → Double DQN.
- (4) Reward magnitude / scale → reward clipping to [-1, 1] or reward normalization.
- (5) Sparse reward → n-step returns or intrinsic curiosity.
- Every DQN implementation combines at least (1) + (2); Rainbow adds all six.
Check yourself — multiple choice
- Random
- Correlation → replay; moving target → target net; overestimation → Double DQN; reward scale → clipping; sparse reward → n-step / curiosity — DQN needs replay + target net minimum
- Not unstable
- Only replay
DQN instability fixes: replay + target net + Double + clipping + n-step / curiosity.
#value-methods#deep-rl#engineering
Practise Reinforcement Learning
214 interview questions in this topic.