Why is subtracting a baseline OK in policy gradient?

hard

Answer

$E[b(s)\; \nabla \operatorname{log}\;{\pi}(a \mid s)]\; = \;0$
$This adds no bias but shrinks variance by centering the reward signal.$
$Choice of baseline: constant, moving average of returns, or state-dependent V(s) (best — cancels state-level variance).$
$Advantage function A(s, a) = Q(s, a) - V(s) is the popular choice \to gives us actor-critic architectures.$

Check yourself — multiple choice

Adds bias
E[b(s) ∇log π] = 0 for b independent of a → no bias, less variance; V(s) is best baseline (cancels state variance) → advantage = actor-critic
Random
Not real

Baseline b(s) has E = 0 in policy gradient → unbiased but lower variance.

#policy-methods#theory

214 interview questions in this topic.