The log-derivative trick — why is it central to policy gradients?
hardAnswer
- For expectation of f under π_θ: ∇_θ E_π[f] = E_π.
- Enables Monte Carlo estimation of gradient using only samples from π and gradient of log-likelihood — no need to differentiate through the reward function or the environment.
- Foundation of REINFORCE, actor-critic, natural gradient, and importance sampling.
- Trick: π_θ(x) ∇_θ log π_θ(x) = ∇_θ π_θ(x).
Check yourself — multiple choice
- Random
- ∇_θ E_π[f] = E_π; enables MC gradient estimation from samples + ∇log π without differentiating through env — foundation of REINFORCE
- Same as MLE
- Not real
Log-derivative: ∇E_π[f] = E_π[f ∇log π]; MC gradient without diff'ing env.
#theory#policy-methods
Practise Reinforcement Learning
214 interview questions in this topic.
Related questions
- Why is subtracting a baseline OK in policy gradient?
- What is natural gradient in policy gradient methods?
- Reparameterization trick in stochastic policies.
- Off-policy policy gradient — how does importance sampling enter?
- Policy gradient as mirror descent — the connection.
- Options framework — formalization.