EasyDeepLearn

Bradley-Terry model in reward modeling.

hard

Answer

  • Probability preferred response yw  >  yly_{w}\; > \;y_{l} given prompt x: P(yw  >  yl  ∣  x)  =  σ(r(x,  yw)  −  r(x,  yl))P(y_{w}\; > \;y_{l}\; \mid \;x)\; = \;{\sigma}(r(x, \;y_{w})\; - \;r(x, \;y_{l})).
  • Loss: -log σ(r(x,  yw)  −  r(x,  yl)){\sigma}(r(x, \;y_{w})\; - \;r(x, \;y_{l})) over preference dataset.
  • Reward model r_φ is scalar-output classifier (usually initialized from SFT model + linear head).
  • Modern reward models: hundreds of thousands of preference pairs; usually much more efficient than expected due to LLM capabilities.
Check yourself — multiple choice
  • Random
  • P(yw  >  yl  ∣  x)  =  σ(r(x,  yw)  −  r(x,  yl))P(y_{w}\; > \;y_{l}\; \mid \;x)\; = \;{\sigma}(r(x, \;y_{w})\; - \;r(x, \;y_{l})); loss = -log σ over preference pairs; r_φ scalar model initialized from SFT + linear head
  • Same as CE
  • Not real

Bradley-Terry: σ(rw  −  rl){\sigma}(r_{w}\; - \;r_{l}) preference model; MLE via -log σ.

#llm#alignment

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions