TD(λ) — how does it work?
hardAnswer
- Interpolates between TD(0) (λ=0, low variance biased) and MC (λ=1, unbiased high variance).
- Uses eligibility traces that decay by γλ each step and increment at visited states.
- Update: V(s) ← V(s) + α δ e(s) for every s.
- Online, single-pass, works with function approximation.
- Foundation of Sutton-Barto RL; GAE is essentially TD(λ) for advantages.
Check yourself — multiple choice
- Random
- Interpolate TD(0) and MC via λ; eligibility traces decay γλ; update V(s) ← V + α δ e(s) online; GAE is TD(λ) for advantages
- Same as MC
- Not real
TD(λ): eligibility traces + λ interpolation between TD(0) and MC.
#theory
Practise Reinforcement Learning
214 interview questions in this topic.