DPO vs PPO for RLHF — practical comparison.
hardAnswer
- PPO: needs reward model + rollouts + KL + value function; complex + unstable.
- DPO: single stage on preference pairs, no RM or PPO, much simpler + more stable.
- Empirically comparable or better than PPO for most tasks (Rafailov 2023).
- Downsides: (1) still needs preference data.
- (2) Off-policy in a sense (uses static preference data).
- (3) Weaker for tasks with objective correctness (math, code) where RL from verifier is better.
- PPO still standard at frontier labs for large models.
Check yourself — multiple choice
- Same
- DPO: single-stage direct preference loss, no RM/PPO, simpler + stable; PPO: complex but still frontier default for large models; DPO weaker on verifiable tasks (math/code)
- Random
- Only DPO
DPO simpler + comparable to PPO on preferences; PPO still frontier default.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.