PPO implementation details that actually matter.
hardAnswer
- Engstrom et al. 2020 'Implementation Matters': (1) value function clipping, (2) reward scaling by running SD, (3) orthogonal init, (4) Adam with fixed lr , (5) global gradient clip to 0.5, (6) GAE with λ = 0.95, (7) normalize advantages per-batch, (8) LR annealing, (9) N epochs per rollout (~4-10).
- Many published gains are due to these tricks, not the core algorithm.
- Always study a modern reference implementation.
Check yourself — multiple choice
- Random
- Value clipping, reward scaling, orthogonal init, gradient clip 0.5, GAE λ=0.95, advantage normalize, LR anneal, 4-10 epochs/rollout — Engstrom 2020 showed these matter
- Not important
- Just clip
PPO tricks: value clip / reward scale / orthogonal init / clip 0.5 / GAE 0.95 / adv norm / anneal / N epochs.
#policy-methods#engineering
Practise Reinforcement Learning
214 interview questions in this topic.