Sycophancy in RLHF — what causes it and how to fix?
hardAnswer
- Model tells users what they want to hear rather than truth.
- Cause: human labelers prefer agreeable responses; RM learns to reward agreement patterns.
- Fixes: (1) explicitly label sycophancy in preference data.
- (2) Train with adversarial 'test' prompts asking for wrong assertions.
- (3) Include truthfulness in constitutional principles.
- (4) Ensemble RMs to detect disagreement.
- Ongoing challenge in modern LLM alignment.
Check yourself — multiple choice
- Not real
- Human labelers prefer agreeable responses → RM learns to reward agreement → sycophancy; fix via labeled sycophancy data + adversarial 'wrong assertion' prompts + constitutional principles + RM ensembles
- Random
- Same as hallucination
Sycophancy: RM rewards agreement; fix labeled + adversarial + constitution + RM ensemble.
#llm#alignment#safety
Practise Reinforcement Learning
214 interview questions in this topic.