Why do many teams choose DPO over PPO for alignment?

medium

Answer

$Mostly engineering cost.$
$PPO-based RLHF requires four models in memory at once, the policy, the reference, the reward model and the critic, plus a generation loop inside training, which makes the pipeline expensive, slow and full of hyperparameters that can silently ruin a run.$
$DPO reformulates preference optimization so the reward model is implicit, turning the problem into a supervised loss on preference pairs, and it needs only the policy and a frozen reference.$
$That makes it dramatically simpler to implement, reproduce and debug.$
$The tradeoff is that DPO learns from a fixed preference set and cannot explore, so it does not benefit from on-policy samples, and at the frontier iterative or online variants that reintroduce fresh sampling tend to close the remaining quality gap.$

Check yourself — multiple choice

DPO is theoretically superior
DPO replaces the reward model and critic with a supervised loss on preference pairs, needing only policy and reference — far simpler, at the cost of no on-policy exploration
PPO does not work on text
They are identical algorithms

DPO trades on-policy exploration for a much simpler pipeline with two models instead of four.

#llm#alignment

214 interview questions in this topic.