How does DPO simplify RLHF?
hardAnswer
- DPO (Rafailov 2023) skips both the reward model and the PPO loop.
- Given preference pairs (prompt, chosen, rejected), directly fine-tune the policy π_θ against a frozen reference with loss: -log sigmoid().
- Equivalent to RLHF optimum under Bradley-Terry preferences.
- Much simpler: one supervised loss, no rollouts, no RM.
- Now the default alignment method for most open LLMs (Zephyr, Tulu, Mistral, Qwen).
Check yourself — multiple choice
- DPO uses PPO internally
- Direct preference optimization: supervised loss on log-ratio between policy and reference — no RM, no PPO
- Only classification
- Same as RLHF
DPO: reparametrize RLHF as a supervised log-sigmoid loss over log-ratios — simpler, cheaper.
#dpo#alignment#preference
Practise LLMs & GenAI
214 interview questions in this topic.