ORPO vs DPO — key difference.
hardAnswer
- DPO: needs reference policy (extra memory + inference cost during training).
- ORPO: no reference policy — uses odds ratio between chosen/rejected.
- Simpler infra (one model instead of two).
- Comparable performance to DPO on many benchmarks.
- Modern choice: ORPO when reference model is inconvenient (larger models, custom architectures) or when you want simpler infra.
Check yourself — multiple choice
- Same thing
- DPO: needs reference policy (memory + inference cost); ORPO: no reference, uses odds-ratio → simpler; comparable performance; ORPO wins on infra simplicity
- Random
- Only DPO
ORPO: no reference policy; simpler than DPO; comparable performance.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.