What is ORPO?
hardAnswer
- ORPO (Hong et al., 2024) merges SFT and preference optimization into a single stage.
- Loss = SFT loss on chosen + λ * odds-ratio penalty encouraging chosen over rejected.
- No reference model needed (unlike DPO), which halves memory.
- Skip SFT entirely and go directly from base to aligned.
- Matches DPO on chat benchmarks in a single-stage recipe — popular for compute-constrained alignment.
Check yourself — multiple choice
- ORPO requires PPO
- Single-stage SFT + odds-ratio preference penalty, no reference model → half the memory of DPO
- Same as RLHF
- Only classification
ORPO: SFT + preference in one stage, no reference → simpler, memory-efficient.
#alignment#preference
Practise LLMs & GenAI
214 interview questions in this topic.