How is PPO adapted for RLHF and what are the main pitfalls?

hard

Answer

${\pi}(a \mid s)$
$\mathrm{Reward}\; = \;\mathrm{RM}(\mathrm{prompt}, \;\mathrm{completion})\; - \;{\beta}\; \cdot \;\operatorname{KL}({\pi}\; \mid \mid \;{\pi}_{\mathrm{SFT}})$
$The KL penalty prevents the policy from drifting so far that RM predictions become unreliable ('over-optimization').$
$Pitfalls: (1) reward hacking — model exploits RM patterns (e.g., always says 'great question!'); (2) mode collapse — outputs become repetitive; (3) instability — PPO is sensitive to β, learning rate, and rollout batch size.$
$In practice: 4-8 PPO epochs per RM.$

Check yourself — multiple choice

PPO ignores the SFT model
Reward = RM(x, y) - β· $\operatorname{KL}({\pi}\; \mid \mid \;{\pi}_{\mathrm{SFT}})$ ; pitfalls: reward hacking, mode collapse, instability
No KL penalty used
PPO doesn't apply to LLMs

RLHF-PPO: reward - β·KL to SFT; pitfalls = reward hacking + collapse + instability.

#rlhf#alignment

214 interview questions in this topic.