RLHF pipeline — the three stages in detail.
hardAnswer
- (1) SFT: fine-tune pretrained LM on curated (prompt, ideal response) demos → gives baseline capable model.
- (2) Reward Modeling: collect preference pairs (prompt, response A vs B, human picks winner); train reward model r_φ(x, y) via Bradley-Terry loss: -log .
- (3) RL: fine-tune π_θ against r_φ with PPO + KL penalty to reference to prevent reward hacking + preserve capabilities.
Check yourself — multiple choice
- Random
- SFT → Reward Model on preference pairs (Bradley-Terry) → PPO fine-tune with KL penalty to ref policy; three-stage foundation of ChatGPT / Claude
- Same as SFT
- Not real
RLHF: SFT + Reward Model (Bradley-Terry) + PPO with KL to ref.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.