Deadly triad — the three ingredients.
hardAnswer
- (1) Function approximation (neural nets, not tabular).
- (2) Bootstrapping (TD target uses V, not full MC).
- (3) Off-policy data.
- Combining all three can cause divergence (Baird's counterexample: linear FA + off-policy TD → weights blow up).
- Modern deep RL usually has all three (DQN, SAC) — works empirically via replay + target nets + gradient clip.
- Understanding this triad is core RL theory.
Check yourself — multiple choice
- Random
- (1) Function approx + (2) Bootstrapping + (3) Off-policy → can diverge (Baird); modern deep RL uses all three but relies on empirical tricks (replay + target + clip)
- Only convergent
- Not real
Deadly triad: FA + bootstrap + off-policy → can diverge; empirically handled.
#theory
Practise Reinforcement Learning
214 interview questions in this topic.