What is 'alignment tax' in RLHF?
hardAnswer
- Fine-tuning for alignment (safety, style) degrades other capabilities.
- E.g., RLHF for helpfulness sometimes hurts factual accuracy or reasoning.
- Root cause: KL to reference policy is not strong enough vs conflicting reward pull; sample distribution during RL doesn't cover all training-distribution capabilities.
- Mitigation: (1) mix pretraining tokens during RL, (2) hold-out benchmark tracking (MMLU, HumanEval), (3) stronger KL to reference, (4) mix RL with continued SFT on diverse data.
Check yourself — multiple choice
- No tax
- Alignment fine-tuning degrades other capabilities (RLHF hurts factual / reasoning); mitigate via mixing pretraining tokens + benchmark monitoring + stronger KL + continued SFT
- Random
- Not real
Alignment tax: RLHF degrades other skills; mix pretraining + monitor + KL.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.