EasyDeepLearn

Why does RLHF include a KL penalty?

hard

Answer

  • PPO fine-tunes against learned reward model r_φ, which is imperfect and prone to reward hacking (model  finds  spurious  features  that  rφ  overestimates)(\mathrm{model}\;\mathrm{finds}\;\mathrm{spurious}\;\mathrm{features}\;\mathrm{that}\;r{\varphi}\;\mathrm{overestimates}).
  • KL penalty β KL⁡(πθ  ∣∣  πref)\operatorname{KL}({\pi}{\theta}\; \mid \mid \;{\pi}_{\mathrm{ref}}) keeps π_θ close to SFT reference → limits how badly model can exploit reward.
  • Also preserves capabilities from pretraining.
  • Choice of β matters: too high → no learning; too low → catastrophic drift.
  • Standard β = 0.01-0.1 for chat models.
Check yourself — multiple choice
  • Random
  • KL⁡(πθ  ∣∣  πref)\operatorname{KL}({\pi}{\theta}\; \mid \mid \;{\pi}_{\mathrm{ref}}) prevents reward hacking + preserves pretraining capabilities; β = 0.01-0.1 typical; too high → no learning, too low → catastrophic drift
  • Same as regularization
  • Not real

KL penalty: prevents reward hacking + preserves capabilities; β 0.01-0.1.

#llm#alignment

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions