In RLHF, why does the reward model become unreliable as training progresses?

hard

Answer

$Because the policy moves off the distribution the reward model was trained on.$
$Preference data came from earlier, weaker samples, so the reward model is accurate there and increasingly extrapolating as the policy improves, which the policy then exploits: it finds text scoring highly under the reward model that humans do not actually prefer.$
$This is why the KL penalty against the reference policy exists, since it keeps the policy inside the region where the reward model was fit rather than expressing a preference for the original model.$
$The operational answers are to monitor the KL divergence as a first-class metric, collect fresh preference data on current samples and retrain the reward model, and keep a held-out human evaluation as the ground truth, because reward going up is not evidence of quality going up.$

Check yourself — multiple choice

Reward models never degrade
The improving policy drifts off the reward model's training distribution and exploits its extrapolations — hence the KL penalty, iterative fresh preference data, and held-out human evaluation
Because the discount factor is wrong
Only the learning rate matters

Reward over-optimization is distributional: the KL term and iterative retraining keep the proxy valid.

#llm#alignment

214 interview questions in this topic.