Describe the full RLHF pipeline in three stages.
hardAnswer
- (1) SFT: fine-tune base LM on instruction data.
- (2) Reward Model: collect preference pairs (prompt, chosen, rejected) from human annotators; train an RM (typically the SFT model with a scalar head) to predict which of two completions humans prefer via Bradley-Terry / log-sigmoid loss.
- (3) RL fine-tune: use PPO to maximize the RM's score with a KL penalty against the SFT reference to prevent reward hacking.
- Output: a preference-aligned model.
- Introduced in InstructGPT (Ouyang 2022).
Check yourself — multiple choice
- One-step SFT
- SFT → train Reward Model on preference pairs → PPO with KL to SFT ref → preference-aligned model
- RLHF has no SFT step
- Only fine-tune with cross-entropy
RLHF: SFT → RM (preference pairs) → PPO with KL against SFT.
#alignment#rlhf
Practise LLMs & GenAI
214 interview questions in this topic.