Off-policy policy gradient — how does importance sampling enter?
hardAnswer
- Data collected under behavior , but want to update target π_θ: use importance weight ρ = π_θ → ∇J ≈ E_π_b[ρ ∇log π_θ · A].
- Weight variance explodes when π_θ far from .
- Solutions: clip ratio (PPO), truncate importance weights (Retrace, V-trace in IMPALA), or use deterministic policy gradient (no IS needed — DDPG/SAC).
Check yourself — multiple choice
- Same as on-policy
- IS weight ρ = π_θ/ corrects for behavior/target mismatch; variance explodes if policies diverge → clip (PPO) / truncate (V-trace, Retrace) / deterministic PG (no IS)
- Random
- Not real
Off-policy PG: IS weight π_θ/; clip / truncate / deterministic PG.
#policy-methods#theory
Practise Reinforcement Learning
214 interview questions in this topic.
Related questions
- Why is subtracting a baseline OK in policy gradient?
- The log-derivative trick — why is it central to policy gradients?
- What is natural gradient in policy gradient methods?
- Reparameterization trick in stochastic policies.
- Policy gradient as mirror descent — the connection.
- Options framework — formalization.