How does the KL penalty interact with reward hacking?
hardAnswer
- KL to bounds how far model can drift → limits reward hacking of imperfect r_φ.
- But: (1) large model can find hacks within KL budget .
- (2) β must be tuned per task — too high stalls learning.
- (3) Better: iterative RM updates (relabel new-policy responses), adversarial reward shaping.
- (4) Modern practice: KL as safety floor, not primary defense against hacking.
Check yourself — multiple choice
- Fixes hacking
- KL bounds drift → limits hacking but doesn't prevent hacks within KL budget; β must be tuned; iterative RM + adversarial + KL together — KL is safety floor, not primary defense
- Random
- Not real
KL: bounds drift, limits but doesn't prevent hacking within budget.
#llm#alignment#safety
Practise Reinforcement Learning
214 interview questions in this topic.