What is the 'alignment tax' and how do you minimize it?
mediumAnswer
- Aligned models often score slightly worse than their base counterpart on capability benchmarks (MMLU, BBH, HumanEval) — the 'alignment tax' — because refusal training + style constraints eat some capability.
- Modern practice minimizes it via: (1) mixing capability-preserving SFT data alongside chat data; (2) KL penalty against the SFT model during RLHF; (3) DPO's log-ratio structure (less drift than PPO); (4) careful data curation to avoid hurting math/code skills.
Check yourself — multiple choice
- Alignment always improves capability
- Aligned models score slightly worse on capability benchmarks; mitigate via mixed SFT, KL penalty, DPO, careful data
- No tradeoff exists
- Only benchmark artifact
Alignment tax: real but small; mitigate via KL, mixed SFT data, careful curation.
#alignment
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is supervised fine-tuning (SFT) and where does it fit in the alignment pipeline?
- How is high-quality instruction-tuning data constructed?
- Describe the full RLHF pipeline in three stages.
- Why does the reward model use a Bradley-Terry / log-sigmoid loss?
- How is PPO adapted for RLHF and what are the main pitfalls?
- How does DPO simplify RLHF?