Interview: 'walk me through the tradeoffs of SFT-only vs SFT+RLHF vs SFT+DPO.'
hardAnswer
- SFT only: fast + simple + reliable + no reward hacking.
- Ceiling limited by demo quality; can't leverage preferences.
- SFT+RLHF (PPO): learns from preferences + can exceed demo quality + slow + expensive + reward hacking risk + complex infra.
- SFT+DPO: preferences without PPO complexity + faster + more stable than PPO + tie infrastructure to reference model; on-par or slightly worse than tuned PPO.
- Modern industry mostly does SFT + DPO / IPO / KTO; frontier labs still PPO.
Check yourself — multiple choice
- Same
- SFT: simple, capped by demos; SFT+RLHF: preferences + exceeds demos + reward hacking + complex; SFT+DPO: preferences + simpler + stable; industry mostly DPO, frontier still PPO
- Random
- Only SFT
SFT vs RLHF vs DPO: SFT simple/capped; PPO powerful/complex; DPO middle ground.
#interview#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.