EasyDeepLearn

What is natural gradient in policy gradient methods?

hard

Answer

  • Follow the direction of steepest ascent in policy space measured by KL divergence (Fisher information metric) instead of Euclidean.
  • Update: θ ← θ + α F(θ)−1F({\theta}) - 1 ∇J where F is Fisher information matrix.
  • Invariant to reparameterization.
  • Foundation of TRPO (approximate) and NPG.
  • Practically: F−1F^{- 1} expensive — use Conjugate Gradient (TRPO) or K-FAC.
  • Underpins why PPO's clipping approximates a trust region.
Check yourself — multiple choice
  • Same as gradient
  • θ ← θ + α F−1F^{- 1} ∇J using Fisher information metric; invariant to reparameterization; F−1F^{- 1} costly → CG (TRPO) or K-FAC; PPO clipping is trust-region approximation
  • Random
  • Not real

Natural gradient: F−1−weightedF^{- 1} - \mathrm{weighted}; reparam-invariant; foundation of TRPO.

#policy-methods#theory

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions