PPO's clipped surrogate objective — write it and explain.

hard

Answer

$L^{\mathrm{CLIP}}({\theta})\; = \;E[\operatorname{min}(r_{t}({\theta})\;A_{t}, \;\mathrm{clip}(r_{t}({\theta}), \;1 - {\varepsilon}, \;1 + {\varepsilon})\;A_{t})]$
${\pi}_{\mathrm{old}}$
$Typical ε = 0.2.$
$Combined with GAE advantage and clipped value loss \to PPO2 (standard).$
$Simpler and works better in practice than TRPO's constrained optimization.$

Check yourself — multiple choice

Random
$L^{\mathrm{CLIP}}\; = \;E[\operatorname{min}(r_{t}\;A_{t}, \;\mathrm{clip}(r_{t}, \;1 - {\varepsilon}, \;1 + {\varepsilon})\;A_{t})]$ with $r_{t}\; = \;{\pi}$ _θ/ ${\pi}_{\mathrm{old}}$ , ε ≈ 0.2; clip caps policy update size; simpler than TRPO's constrained opt
Same as REINFORCE
Not real

PPO: $L^{\mathrm{CLIP}}\; = \;E[\operatorname{min}(\mathrm{rA}, \;\mathrm{clip}(r, \;1 - {\varepsilon}, \;1 + {\varepsilon})\;A)]$ ; ε ≈ 0.2.

#policy-methods#actor-critic

214 interview questions in this topic.