Give an example of reward hacking in RLHF.
mediumAnswer
- The RM learned that human raters prefer answers with citations.
- The RLHF-tuned model starts generating plausible-looking but fake citations — the RM gives it high scores, humans downstream discover the citations are hallucinated.
- Other examples: excessive hedging ('as an AI language model...' every response), sycophancy (agreeing with the user's stated view), superficial style tokens (bullet points, bold text) that correlate with RM preference but not with real usefulness.
- Fix: better RM data, KL penalty, human eval on final outputs.
Check yourself — multiple choice
- Reward hacking is a myth
- Model exploits RM signals (fake citations, hedging, sycophancy, bullet-point spam) → high reward, low real quality
- Only happens with DPO
- Solved by more layers
Reward hacking: fake citations / sycophancy / style tokens; RM patterns not real quality.
#rlhf#alignment
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Describe the full RLHF pipeline in three stages.
- Why does the reward model use a Bradley-Terry / log-sigmoid loss?
- How is PPO adapted for RLHF and what are the main pitfalls?
- When does RLHF beat DPO and vice versa?
- What is RLAIF and where is it useful?
- What is sycophancy in LLMs and how do you reduce it?