24 interview questions on rlhf & llm alignment, each answered in full. Free to read, no account needed.
What is RLHF and how does it fit into training an LLM?
hard- RLHF (Reinforcement Learning from Human Feedback) fine-tunes a pretrained language model to align with human preferences.
- Steps: (1) supervised fine-tuning on curated demonstrations; (2) train a reward model on human preference pairs; (3) optimize the LLM policy against the reward model using PPO, with a KL penalty to stay close to the reference policy.
- Recent alternatives: DPO removes the RL step by optimizing preferences directly.
RLHF pipeline — the three stages in detail.
hard- (1) SFT: fine-tune pretrained LM on curated (prompt, ideal response) demos → gives baseline capable model.
- (2) Reward Modeling: collect preference pairs (prompt, response A vs B, human picks winner); train reward model r_φ(x, y) via Bradley-Terry loss: -log σ(r(x,yw)−r(x,yl)).
- (3) RL: fine-tune π_θ against r_φ with PPO + KL penalty to reference πref (βKL(πθ∣∣πref)) to prevent reward hacking + preserve capabilities.
Bradley-Terry model in reward modeling.
hard- Probability preferred response yw>yl given prompt x: P(yw>yl∣x)=σ(r(x,yw)−r(x,yl)).
- Loss: -log σ(r(x,yw)−r(x,yl)) over preference dataset.
- Reward model r_φ is scalar-output classifier (usually initialized from SFT model + linear head).
- Modern reward models: hundreds of thousands of preference pairs; usually much more efficient than expected due to LLM capabilities.
Why does RLHF include a KL penalty?
hard- PPO fine-tunes against learned reward model r_φ, which is imperfect and prone to reward hacking (modelfindsspuriousfeaturesthatrφoverestimates).
- KL penalty β KL(πθ∣∣πref) keeps π_θ close to SFT reference → limits how badly model can exploit reward.
- Also preserves capabilities from pretraining.
- Choice of β matters: too high → no learning; too low → catastrophic drift.
- Standard β = 0.01-0.1 for chat models.
DPO vs PPO for RLHF — practical comparison.
hard- PPO: needs reward model + rollouts + KL + value function; complex + unstable.
- DPO: single stage on preference pairs, no RM or PPO, much simpler + more stable.
- Empirically comparable or better than PPO for most tasks (Rafailov 2023).
- Downsides: (1) still needs preference data.
- (2) Off-policy in a sense (uses static preference data).
- (3) Weaker for tasks with objective correctness (math, code) where RL from verifier is better.
- PPO still standard at frontier labs for large models.
RLAIF — RL from AI Feedback.
hard- Replace human labelers with LLM judge for preference labels.
- Bai et al. 2022 (Constitutional AI): iteratively use a strong LLM to critique + revise responses per a written constitution.
- Advantages: scale (billions of preferences), consistency, cost.
- Trade-offs: risk of amplifying LLM biases, less alignment with real user preferences.
- Standard in modern alignment (Anthropic's Constitutional AI, Meta's Llama, Google's Gemini all use RLAIF + human).
RL for reasoning (o1, DeepSeek-R1) — the paradigm shift.
hard- Train LLM to generate long chain-of-thought reasoning by RL with automatic reward from verifier (math answer correct, code passes tests).
- Model learns to think longer, retry, self-correct.
- DeepSeek-R1: pure RL from base model (no SFT step) → emergent reasoning behaviors.
- Uses GRPO (group-normalized rewards).
- Test-time compute becomes a knob.
- Marks shift from imitation-based to verification-based post-training.
How does RL train chain-of-thought reasoning?
hard- Reward on FINAL answer only (verifier: math correct? code passes tests?).
- Intermediate tokens get gradient via policy gradient on the final reward, credit-assigned across the response.
- Long CoT emerges because: (1) longer thinking gives higher chance of correct answer, (2) reward differential per token pushes toward exploration of reasoning traces.
- Combined with KL penalty to preserve language + prevent gibberish → emergent step-by-step reasoning.
Verifier reward vs preference-based reward — when to use each?
medium- Verifier: objective correctness (math, code, structured tasks) — automatic, cheap, unambiguous, scalable.
- Best when correctness is well-defined.
- Preference: subjective quality (helpfulness, tone, style, creative writing) — needs human/AI labels, more ambiguous.
- Best when 'right' is unclear.
- Modern LLM training combines both: verifier RL for skills (o1 reasoning), preference RLHF for chat quality, safety.
Process reward vs outcome reward for reasoning.
hard- Outcome reward: only final answer scored.
- Simple but sparse — credit assignment across long CoT is hard.
- Process reward (PRM, Lightman et al. 2023): score each reasoning step for correctness.
- Denser signal, better for long CoT.
- Training PRMs requires per-step labels (expensive).
- Modern practice: mix outcome + process; auto-generate process labels via LLM verifier or Monte Carlo (reason forward from step, check outcome distribution).
Iterative RLHF — why do you keep going?
hard- Single-round RLHF: preference data collected on SFT model → RM trained → policy trained.
- Problem: new policy visits states RM was never trained on → RM predictions drift.
- Fix: iterative rounds — collect new preferences on current-policy responses, update RM, redo PPO/DPO.
- Modern frontier LLMs go through many rounds.
- Cost: expensive labeler cycles.
- Mitigation: mix human + RLAIF preferences.
What can RL do that SFT cannot?
medium- SFT does behavior cloning of ideal responses.
- RL adds: (1) preference between responses (SFT can't compare).
- (2) Learning from FAILED attempts (verifier RL, HER-style relabeling).
- (3) Exploration of the response space beyond demos.
- (4) Direct optimization of test-time desirable behavior (long CoT, tool use, safety).
- (5) Scaling with test-time compute.
- Rule: SFT gives baseline capability; RL adds targeted post-training improvements.
How do reward models scale with size?
hard- Larger RMs (closer in size to policy) give better preferences: better calibration, less easy-to-exploit gaps.
- But: too small → doesn't understand nuance; too large → expensive per training step.
- Modern: RM around 1/3 of policy size (7B RM for 22B policy).
- Ensemble of RMs improves robustness against hacking.
- Some frontier labs use LLM-as-judge (whole LLM) instead of scalar-output RM head.
How do you fix length bias in RLHF?
medium- (1) Length-normalize reward: divide by response length (log or square-root).
- (2) Add length penalty λ · length to policy loss.
- (3) In DPO: length-normalize log-probs.
- (4) Curate preference pairs to be length-matched.
- (5) Train RM to explicitly discount length (RM sees length as feature and is penalized for using it).
- Standard fix in modern LLM training; without it, models become verbose.
ORPO — Odds Ratio Preference Optimization.
hard- Hong et al. 2024.
- Combines SFT and preference optimization in one loss without a reference model.
- Loss = SFT loss + λ · LOR where LOR is preference log odds ratio: -log σ(log(odds(yw)/odds(yl))).
- No reference policy needed, no separate SFT stage.
- Simpler pipeline than SFT + DPO.
- Standard for open-source model alignment when reference model is unavailable / undesired.
KTO — Kahneman-Tversky Optimization.
hard- Ethayarajh et al. 2024.
- Uses prospect theory (loss aversion): people care more about avoiding losses than gaining.
- Works with unpaired (binary desirable/undesirable) labels — easier to collect than paired preferences.
- Loss makes desirable outputs more likely + undesirable less likely, weighted asymmetrically.
- Alternative to DPO when preference pairs are hard to get.
What is 'alignment tax' in RLHF?
hard- Fine-tuning for alignment (safety, style) degrades other capabilities.
- E.g., RLHF for helpfulness sometimes hurts factual accuracy or reasoning.
- Root cause: KL to reference policy is not strong enough vs conflicting reward pull; sample distribution during RL doesn't cover all training-distribution capabilities.
- Mitigation: (1) mix pretraining tokens during RL, (2) hold-out benchmark tracking (MMLU, HumanEval), (3) stronger KL to reference, (4) mix RL with continued SFT on diverse data.
How does RL train tool-use in LLM agents?
hard- (1) Reward on task success (e.g., verifier checks correct API call result).
- (2) Multi-step trajectories with intermediate tool calls.
- (3) GRPO / PPO with KL penalty.
- (4) Environments: WebArena, ALFWorld, SWE-bench.
- (5) Format constraints: tool call syntax must be valid → additional reward shaping.
- Modern: agentic RL is next frontier of LLM training.
- Reward hacking risk: model finds shortcuts (e.g., fake API responses).
Interview: 'walk me through the tradeoffs of SFT-only vs SFT+RLHF vs SFT+DPO.'
hard- SFT only: fast + simple + reliable + no reward hacking.
- Ceiling limited by demo quality; can't leverage preferences.
- SFT+RLHF (PPO): learns from preferences + can exceed demo quality + slow + expensive + reward hacking risk + complex infra.
- SFT+DPO: preferences without PPO complexity + faster + more stable than PPO + tie infrastructure to reference model; on-par or slightly worse than tuned PPO.
- Modern industry mostly does SFT + DPO / IPO / KTO; frontier labs still PPO.
ORPO vs DPO — key difference.
hard- DPO: needs reference policy πref (extra memory + inference cost during training).
- ORPO: no reference policy — uses odds ratio between chosen/rejected.
- Simpler infra (one model instead of two).
- Comparable performance to DPO on many benchmarks.
- Modern choice: ORPO when reference model is inconvenient (larger models, custom architectures) or when you want simpler infra.
LLM agent benchmarks — what tests capability?
medium- (1) WebArena / VisualWebArena: browse web + execute tasks.
- (2) ALFWorld: navigate textual household environment.
- (3) SWE-bench: fix real GitHub issues.
- (4) AgentBench: comprehensive tool-use.
- (5) OSWorld: real OS interaction.
- (6) τ-bench: customer-service dialog tasks.
- Rule: test in realistic settings (not just multiple choice); track success rate + cost per task + safety violations.
- Modern agent RL trained on some of these.
'Verifier + refiner' RL pattern for reasoning.
hard- (1) Sample many reasoning chains from model.
- (2) Verifier (external tool or LLM judge) scores each.
- (3) Best-of-N: keep highest scoring — cheap test-time scaling.
- (4) RL fine-tune: train model to produce chains verifier likes.
- (5) Iterative: use fine-tuned model to generate next round of chains, retrain.
- Foundation of o1, DeepSeek-R1, and many modern reasoning models.
- Test-time compute becomes tunable.
In RLHF, why does the reward model become unreliable as training progresses?
hard- Because the policy moves off the distribution the reward model was trained on.
- Preference data came from earlier, weaker samples, so the reward model is accurate there and increasingly extrapolating as the policy improves, which the policy then exploits: it finds text scoring highly under the reward model that humans do not actually prefer.
- This is why the KL penalty against the reference policy exists, since it keeps the policy inside the region where the reward model was fit rather than expressing a preference for the original model.
- The operational answers are to monitor the KL divergence as a first-class metric, collect fresh preference data on current samples and retrain the reward model, and keep a held-out human evaluation as the ground truth, because reward going up is not evidence of quality going up.
Why do many teams choose DPO over PPO for alignment?
medium- Mostly engineering cost.
- PPO-based RLHF requires four models in memory at once, the policy, the reference, the reward model and the critic, plus a generation loop inside training, which makes the pipeline expensive, slow and full of hyperparameters that can silently ruin a run.
- DPO reformulates preference optimization so the reward model is implicit, turning the problem into a supervised loss on preference pairs, and it needs only the policy and a frozen reference.
- That makes it dramatically simpler to implement, reproduce and debug.
- The tradeoff is that DPO learns from a fixed preference set and cannot explore, so it does not benefit from on-policy samples, and at the frontier iterative or online variants that reintroduce fresh sampling tend to close the remaining quality gap.