Tabular vs function-approximation RL — the challenge.
hardAnswer
- Tabular: one entry per (s, a) — convergence guaranteed but doesn't scale.
- Function approximation (neural nets): scales to huge state spaces but breaks convergence guarantees.
- The 'deadly triad': off-policy + function approximation + bootstrapping → potential divergence.
- Modern deep RL uses tricks (target networks, replay, gradient clipping) to work despite the triad.
- Understanding this trap is key.
Check yourself — multiple choice
- No issue
- Tabular: guaranteed convergence, doesn't scale. Function approx: scales but breaks guarantees. Deadly triad = off-policy + function approx + bootstrap → potential divergence; mitigated by target nets / replay / clipping
- Random
- Same thing
Deadly triad: off-policy + FA + bootstrapping → divergence risk.
#theory#deep-rl
Practise Reinforcement Learning
214 interview questions in this topic.