When does RLHF beat DPO and vice versa?
hardAnswer
- DPO advantages: much simpler, stable, cheap, no RM.
- Works well when preference data is high quality and the base model is already close to aligned.
- RLHF advantages: RM can be reused across policies; PPO can leverage more preference signal indirectly (online rollouts collect on-policy samples the RM scores).
- Frontier labs (Anthropic, OpenAI) still use RLHF at scale.
- Best practice: SFT + DPO for most open LLMs; add iterative DPO or RLHF-style RM+PPO for the last mile.
Check yourself — multiple choice
- DPO always wins
- DPO: simple, cheap, stable, good for open LLMs. RLHF: reusable RM, on-policy signal — used by frontier labs for the last mile.
- RLHF always wins
- Neither exists
DPO for open LLMs; RLHF still edge-of-frontier for reusable RM + on-policy fine-tuning.
#dpo#rlhf#alignment
Practise LLMs & GenAI
214 interview questions in this topic.