EasyDeepLearn

Interview: 'explain why PPO is on-policy but uses a ratio to importance-sample.'

hard

Answer

  • On-policy in spirit: PPO uses only recent data collected under πold{\pi}_{\mathrm{old}} (close  to  πθ)(\mathrm{close}\;\mathrm{to}\;{\pi}{\theta}).
  • The importance ratio r = π_θ / πold{\pi}_{\mathrm{old}} accounts for the fact that after several gradient steps within the epoch, π_θ has drifted from the data-generating πold{\pi}_{\mathrm{old}} — but not far (clipping enforces this).
  • Compare: fully off-policy (SAC, Q-learning) reuses ancient data.
  • PPO's 'small IS correction' is the whole point of the clipped surrogate.
Check yourself — multiple choice
  • Random
  • On-policy in spirit — only recent data; ratio π_θ/πold{\pi}_{\mathrm{old}} accounts for drift within epochs (kept small by clip); fully off-policy (SAC/Q-learning) reuses ancient data — PPO's clipped IS is core mechanism
  • Same as SAC
  • Not real

PPO on-policy w/ small IS correction; clip keeps ratio in trust region.

#interview#policy-methods

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions