GAE — Generalized Advantage Estimation formula.
hardAnswer
- ^∞ where . λ = 0 → TD(0) advantage (biased, low var). λ = 1 → MC advantage (unbiased, high var).
- Typical λ = 0.95 in PPO.
- Provides smooth bias-variance tradeoff.
- Compute efficiently backward from end of trajectory.
- Standard advantage estimator in modern policy gradient methods.
Check yourself — multiple choice
- Random
- with δ = r + γV(s') - V(s); λ=0 → TD(0) biased low-var, λ=1 → MC unbiased high-var; PPO uses λ ≈ 0.95
- Same as TD
- Not real
GAE(λ): weighted sum of TD errors; PPO default λ ≈ 0.95.
#theory#actor-critic
Practise Reinforcement Learning
214 interview questions in this topic.