Iterative RLHF — why do you keep going?
hardAnswer
- Single-round RLHF: preference data collected on SFT model → RM trained → policy trained.
- Problem: new policy visits states RM was never trained on → RM predictions drift.
- Fix: iterative rounds — collect new preferences on current-policy responses, update RM, redo PPO/DPO.
- Modern frontier LLMs go through many rounds.
- Cost: expensive labeler cycles.
- Mitigation: mix human + RLAIF preferences.
Check yourself — multiple choice
- Not needed
- Single round: RM trained on SFT-model responses, drifts when policy changes; iterative rounds relabel on current-policy outputs → updated RM → new PPO/DPO; expensive but essential at frontier
- Random
- Same as one-shot
Iterative RLHF: relabel current-policy outputs; multiple rounds; standard at frontier.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.