ORPO — Odds Ratio Preference Optimization.
hardAnswer
- Hong et al. 2024.
- Combines SFT and preference optimization in one loss without a reference model.
- Loss = SFT loss + λ · where is preference log odds ratio: -log σ().
- No reference policy needed, no separate SFT stage.
- Simpler pipeline than SFT + DPO.
- Standard for open-source model alignment when reference model is unavailable / undesired.
Check yourself — multiple choice
- Same as DPO
- SFT loss + preference log-odds-ratio loss in one step; no reference model needed; simpler than SFT + DPO; open-source alignment when ref model unavailable
- Random
- Not real
ORPO: SFT + odds-ratio preference in one loss; no ref model.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.