What is the deadly triad and how do practical algorithms cope with it?
hardAnswer
- The combination of function approximation, bootstrapping, and off-policy learning.
- Each is fine alone, but together they can make value estimates diverge, because a bootstrapped target is computed from the same approximator being updated, and off-policy data means the states are not distributed according to the policy whose values you are fitting.
- Deep Q-learning uses all three, which is why it needs the stabilizers it has: a target network to freeze the bootstrap target for a while, a replay buffer to decorrelate updates, and reward or gradient clipping to bound the magnitude.
- Double Q-learning additionally removes the maximization bias that makes divergence more likely.
- None of these are cosmetic; removing any one of them typically makes training visibly unstable.
Check yourself — multiple choice
- Three exploration strategies
- Function approximation, bootstrapping and off-policy learning together can diverge — hence target networks, replay buffers, clipping and double Q-learning
- A reward shaping technique
- Three types of policy gradient
The triad explains why DQN's stabilizers exist rather than being optional tricks.
#deep-rl#theory
Practise Reinforcement Learning
214 interview questions in this topic.