Why does the reward model use a Bradley-Terry / log-sigmoid loss?

hard

Answer

$Preference data is pairwise: 'chosen better than rejected'.$
$Bradley-Terry models P(chosen > rejected) = sigmoid(r(chosen) - r(rejected)).$
$\operatorname{sigmoid}(r(\mathrm{chosen})\; - \;r(\mathrm{rejected}))$
$Absolute reward scale is unidentifiable — that's fine: PPO only uses relative rewards.$
$Alternative: DPO reparametrizes this loss directly on the policy, skipping the RM.$

Check yourself — multiple choice

MSE on absolute reward
Bradley-Terry: -log σ(r(chosen) - r(rejected)) — pairwise preference, scale-invariant
L1 loss
Same as cross-entropy

BT loss: -log ${\sigma}(r_{\mathrm{chosen}}\; - \;r_{\mathrm{rejected}})$ — pairwise, scale-invariant, standard for RMs.

#rlhf#alignment

214 interview questions in this topic.