EasyDeepLearn

DPO derivation — the key mathematical insight.

hard

Answer

  • The optimal policy under KL-constrained reward max is π⋅(y∣x){\pi} \cdot (y \mid x) ∝ πref(y∣x){\pi}_{\mathrm{ref}}(y \mid x) exp⁡(r(x,  y)/β)\operatorname{exp}(r(x, \;y) / {\beta}).
  • Invert: r(x, y) = β log⁡(π⋅(y∣x)/πref(y∣x))  +  const\operatorname{log}({\pi} \cdot (y \mid x) / {\pi}_{\mathrm{ref}}(y \mid x))\; + \;\mathrm{const}.
  • Substitute into Bradley-Terry preference model → loss depends only on π_θ, not r: L(θ) = -E[log⁡  σ(β  log⁡(πθ(yw∣x)/πref(yw∣x))  −  β  log⁡(πθ(yl∣x)/πref(yl∣x)))]E[\operatorname{log}\;{\sigma}({\beta}\;\operatorname{log}({\pi}{\theta}(y_{w} \mid x) / {\pi}_{\mathrm{ref}}(y_{w} \mid x))\; - \;{\beta}\;\operatorname{log}({\pi}{\theta}(y_{l} \mid x) / {\pi}_{\mathrm{ref}}(y_{l} \mid x)))].
  • Skips reward model + PPO entirely; direct MLE on preference data.
Check yourself — multiple choice
  • Random
  • Optimal KL-constrained policy has closed-form ∝ πref{\pi}_{\mathrm{ref}} exp⁡(r/β)\operatorname{exp}(r / {\beta}); invert to r = β log⁡(π⋅/πref)\operatorname{log}({\pi} \cdot / {\pi}_{\mathrm{ref}}); substitute in Bradley-Terry → direct loss on π_θ; skips RM + PPO
  • Same as PPO
  • Not real

DPO: closed-form KL-optimal π ∝ πref{\pi}_{\mathrm{ref}} exp⁡(r/β)\operatorname{exp}(r / {\beta}); invert r; direct preference loss.

#llm#alignment#theory

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions