TRPO — Trust Region Policy Optimization.

hard

Answer

$E[{\pi}{\theta} / {\pi}_{\mathrm{old}}\; \cdot \;A]$
$E[\operatorname{KL}({\pi}_{\mathrm{old}}\; \mid \mid \;{\pi}{\theta})]$
$Solved via conjugate gradient + line search on the natural gradient direction.$
$Guarantees monotonic policy improvement (theoretically).$
$Complex to implement + expensive per update.$
$PPO simplifies this to a clipped objective (much easier, similar performance) \to PPO is basically TRPO's practical successor.$

Check yourself — multiple choice

Same as PPO
max $E[{\pi}{\theta} / {\pi}_{\mathrm{old}}\; \cdot \;A]$ s.t. $E[\operatorname{KL}({\pi}_{\mathrm{old}}\; \mid \mid \;{\pi}{\theta})]$ ≤ δ; conjugate gradient + line search on natural gradient; monotone improvement; simplified into PPO clipping
Random
Not real

TRPO: constrained KL trust region + natural gradient; PPO's rigorous predecessor.

#policy-methods#actor-critic

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lesson Flashcards Take the quiz

Related questions