What is iterative DPO / online DPO?
hardAnswer
- Standard DPO uses a fixed set of preference pairs.
- Iterative DPO: after DPO fine-tuning, generate new completions from the updated model, get preferences (from a judge LLM or humans), and DPO-fine-tune again.
- Repeat.
- Each round moves the reference model forward.
- Improves quality closer to full RLHF at a fraction of the compute.
- Zephyr-Beta, Tulu-3, and Llama-3-Instruct use iterative preference optimization variants.
Check yourself — multiple choice
- One shot only
- Repeatedly generate + re-annotate preferences + DPO-fine-tune → closer to RLHF quality at less compute (Zephyr, Tulu-3, Llama-3-Instruct)
- Same as SFT
- Impossible to iterate
Iterative DPO: multi-round preference collection + DPO → strong alignment at lower cost.
#dpo#alignment#preference
Practise LLMs & GenAI
214 interview questions in this topic.