What is the advantage function?
mediumAnswer
- A(s, a) = Q(s, a) - V(s) → how much better is action a than the average action in state s.
- Reduces variance in policy-gradient estimates (baseline subtraction preserves the gradient in expectation but shrinks variance).
- Standard estimators: TD-residual A ≈ r + γ V(s') - V(s) (biased low variance); n-step or GAE for bias-variance tradeoff.
- Used in A2C, PPO, TRPO.
Check yourself — multiple choice
- Same as Q
- A(s, a) = Q(s, a) - V(s); baseline shrinks variance without biasing gradient; TD-residual or GAE estimator; used in A2C / PPO
- Random
- Not real
Advantage A = Q - V; reduces policy-gradient variance.
#theory#actor-critic
Practise Reinforcement Learning
214 interview questions in this topic.