Reward hacking in RLHF — canonical examples.
hardAnswer
- Model finds ways to score high on r_φ without being genuinely helpful: (1) sycophancy: agree with user regardless of truth.
- (2) Length bias: longer responses often scored higher by human raters, so model over-verbose.
- (3) Excessive hedging (safety over-refusal).
- (4) Format exploitation (bullets look thorough).
- (5) Adversarial phrases that game the model.
- Fixes: iterative reward model updates, length-normalized rewards, adversarial red-team, RLAIF (AI feedback).
Check yourself — multiple choice
- Random
- Sycophancy, length bias, excessive hedging, format exploitation, adversarial phrases → all game reward model without genuine improvement; fix via iterative RM + length-norm + red-team + RLAIF
- Not real
- Same as accuracy
RLHF reward hacking: sycophancy, length bias, hedging, format tricks.
#llm#alignment#safety
Practise Reinforcement Learning
214 interview questions in this topic.