DPO — Direct Preference Optimization instead of RLHF PPO.

hard

Answer

$Rafailov et al. 2023: skips explicit reward model + PPO.$
$E[\operatorname{log}\;{\sigma}({\beta}\;\operatorname{log}\;{\pi}{\theta}(y_{w} \mid x) / {\pi}_{\mathrm{ref}}(y_{w} \mid x)\; - \;{\beta}\;\operatorname{log}\;{\pi}{\theta}(y_{l} \mid x) / {\pi}_{\mathrm{ref}}(y_{l} \mid x))]$
$Derived from KL-constrained reward-maximization \to closed-form optimal policy in terms of reward \to invert to remove reward.$
$Simpler + more stable than PPO.$
$Now standard for open-source LLM alignment (Llama, Zephyr).$

Check yourself — multiple choice

Same as PPO
Direct loss on preference pairs; skips reward model + PPO; derived from KL-constrained optimal policy — simpler + more stable; standard for open-source LLM alignment
Random
Not real

DPO: direct preference loss; no reward model + PPO; simpler LLM alignment.

#llm#alignment#policy-methods

214 interview questions in this topic.