Policy gradient as mirror descent — the connection.
hardAnswer
- Policy gradient can be viewed as mirror descent on the policy simplex with the KL divergence as Bregman divergence.
- Natural gradient = mirror descent step in log-policy space.
- TRPO's KL constraint is explicit mirror-descent step.
- This unification (Kakade & Langford, later Vieillard et al.) clarifies why entropy regularization, KL constraints, and log-space parameterization all improve stability.
Check yourself — multiple choice
- Random
- PG = mirror descent on policy simplex with KL as Bregman divergence; natural gradient ≡ mirror descent in log-π space; TRPO's KL constraint is explicit mirror step
- Same as SGD
- Not real
PG ≡ mirror descent with KL Bregman divergence; TRPO explicit mirror step.
#theory#policy-methods
Practise Reinforcement Learning
214 interview questions in this topic.
Related questions
- Why is subtracting a baseline OK in policy gradient?
- The log-derivative trick — why is it central to policy gradients?
- What is natural gradient in policy gradient methods?
- Reparameterization trick in stochastic policies.
- Off-policy policy gradient — how does importance sampling enter?
- Options framework — formalization.