What is the clipping in PPO actually protecting you from?
hardAnswer
- From a policy update so large that the data you collected no longer describes the policy you now have.
- Policy gradient estimates are only valid near the behaviour policy, so a big step invalidates the very samples that justified it, and the classic symptom is a policy that collapses after a promising run.
- Clipping the probability ratio removes the incentive to move a given action's probability beyond a trust region, so the objective flattens instead of rewarding ever larger changes.
- It is a cheap surrogate for the constrained optimization TRPO solves exactly, trading theoretical guarantees for a first-order method that fits in a few lines.
- It does not prevent divergence on its own, which is why implementations also limit epochs per batch and monitor the KL divergence.
Check yourself — multiple choice
- It prevents overfitting to states
- It stops updates so large that the collected data no longer represents the current policy, acting as a cheap trust region in place of TRPO's exact constraint
- It clips the reward signal
- It replaces the value function
Clipping approximates a trust region, keeping the update inside the region where the sampled data is valid.
#policy-methods#actor-critic
Practise Reinforcement Learning
214 interview questions in this topic.