20 interview questions on alignment: sft, rlhf, dpo, each answered in full. Free to read, no account needed.
Describe the full RLHF pipeline in three stages.
hard- (1) SFT: fine-tune base LM on instruction data.
- (2) Reward Model: collect preference pairs (prompt, chosen, rejected) from human annotators; train an RM (typically the SFT model with a scalar head) to predict which of two completions humans prefer via Bradley-Terry / log-sigmoid loss.
- (3) RL fine-tune: use PPO to maximize the RM's score with a KL penalty against the SFT reference to prevent reward hacking.
- Output: a preference-aligned model.
- Introduced in InstructGPT (Ouyang 2022).
Why does the reward model use a Bradley-Terry / log-sigmoid loss?
hard- Preference data is pairwise: 'chosen better than rejected'.
- Bradley-Terry models P(chosen > rejected) = sigmoid(r(chosen) - r(rejected)).
- The RM loss is -log sigmoid(r(chosen)−r(rejected)), which pushes r(chosen) above r(rejected) by a margin.
- Absolute reward scale is unidentifiable — that's fine: PPO only uses relative rewards.
- Alternative: DPO reparametrizes this loss directly on the policy, skipping the RM.
How is PPO adapted for RLHF and what are the main pitfalls?
hard- PPO treats the LLM as a policy π(a∣s), where s=prompt, a=generated tokens.
- Reward=RM(prompt,completion)−β⋅KL(π∣∣πSFT).
- The KL penalty prevents the policy from drifting so far that RM predictions become unreliable ('over-optimization').
- Pitfalls: (1) reward hacking — model exploits RM patterns (e.g., always says 'great question!'); (2) mode collapse — outputs become repetitive; (3) instability — PPO is sensitive to β, learning rate, and rollout batch size.
- In practice: 4-8 PPO epochs per RM.
How does DPO simplify RLHF?
hard- DPO (Rafailov 2023) skips both the reward model and the PPO loop.
- Given preference pairs (prompt, chosen, rejected), directly fine-tune the policy π_θ against a frozen reference πref with loss: -log sigmoid(β⋅(logπθ(chosen∣x)−logπref(chosen∣x))−β⋅(logπθ(rejected∣x)−logπref(rejected∣x))).
- Equivalent to RLHF optimum under Bradley-Terry preferences.
- Much simpler: one supervised loss, no rollouts, no RM.
- Now the default alignment method for most open LLMs (Zephyr, Tulu, Mistral, Qwen).
When does RLHF beat DPO and vice versa?
hard- DPO advantages: much simpler, stable, cheap, no RM.
- Works well when preference data is high quality and the base model is already close to aligned.
- RLHF advantages: RM can be reused across policies; PPO can leverage more preference signal indirectly (online rollouts collect on-policy samples the RM scores).
- Frontier labs (Anthropic, OpenAI) still use RLHF at scale.
- Best practice: SFT + DPO for most open LLMs; add iterative DPO or RLHF-style RM+PPO for the last mile.
What is KTO and when is it useful?
hard- KTO (Kahneman-Tversky Optimization; Ethayarajh 2024) is a DPO variant that uses only binary 'good/bad' feedback per example (not paired preferences) and asymmetric utility weights (losses hurt more than gains — Kahneman-Tversky prospect theory).
- Practically useful when you have thumbs-up / thumbs-down user feedback (much more common than paired A/B preference) — no need to construct chosen/rejected pairs.
- Matches DPO quality on many benchmarks.
What is ORPO?
hard- ORPO (Hong et al., 2024) merges SFT and preference optimization into a single stage.
- Loss = SFT loss on chosen + λ * odds-ratio penalty encouraging chosen over rejected.
- No reference model needed (unlike DPO), which halves memory.
- Skip SFT entirely and go directly from base to aligned.
- Matches DPO on chat benchmarks in a single-stage recipe — popular for compute-constrained alignment.
What is RLAIF and where is it useful?
hard- Reinforcement Learning from AI Feedback (Bai et al., 2022 — Constitutional AI): replace human preference labelers with a stronger LLM asked to rank two completions using a set of guidelines ('the constitution').
- Scales preference data cheaply — millions of AI-rated pairs vs thousands of human-rated ones.
- Works surprisingly well: RLAIF-trained models are competitive with RLHF-trained ones on chat benchmarks.
- Anthropic's Claude uses RLAIF at scale.
What is Constitutional AI?
hard- Anthropic's alignment framework (Bai et al., 2022): the model is given a list of principles ('the constitution' — e.g., 'be helpful and harmless') and asked to critique + rewrite its own outputs when they violate these principles.
- The rewrites become preference pairs used to train a reward model or run RLAIF.
- Reduces human labeling burden and makes safety principles explicit / auditable.
- Foundation of Claude's training.
Give an example of reward hacking in RLHF.
medium- The RM learned that human raters prefer answers with citations.
- The RLHF-tuned model starts generating plausible-looking but fake citations — the RM gives it high scores, humans downstream discover the citations are hallucinated.
- Other examples: excessive hedging ('as an AI language model...' every response), sycophancy (agreeing with the user's stated view), superficial style tokens (bullet points, bold text) that correlate with RM preference but not with real usefulness.
- Fix: better RM data, KL penalty, human eval on final outputs.
What is sycophancy in LLMs and how do you reduce it?
medium- The model changes its answer to match the user's stated opinion or emotional cue, even when wrong ('Are you sure?
- Actually the earth is flat').
- Introduced during RLHF because human raters reward agreeable responses.
- Reductions: (1) RM training data that penalizes agreement without evidence; (2) 'debate' or self-consistency prompting to force the model to check its answer; (3) tool-use (retrieval, code execution) to verify claims; (4) explicit anti-sycophancy in system prompts.
How is 'refusal' behavior trained into LLMs?
medium- SFT data explicitly includes examples where the user asks for something harmful / disallowed and the assistant refuses with a helpful explanation.
- Preference data marks 'good refusals' over 'harmful compliance' and over 'over-refusal'.
- RLHF/DPO stages then further shape refusal.
- Trade-off: too much refusal training → over-refusal ('I can't help with that' for benign queries); too little → jailbreaks.
- Frontier labs iterate this trade-off constantly.
What is the 'alignment tax' and how do you minimize it?
medium- Aligned models often score slightly worse than their base counterpart on capability benchmarks (MMLU, BBH, HumanEval) — the 'alignment tax' — because refusal training + style constraints eat some capability.
- Modern practice minimizes it via: (1) mixing capability-preserving SFT data alongside chat data; (2) KL penalty against the SFT model during RLHF; (3) DPO's log-ratio structure (less drift than PPO); (4) careful data curation to avoid hurting math/code skills.
What is 'honesty' as an alignment target and how do you train for it?
medium- Honesty means the model expresses calibrated uncertainty and refuses to fabricate.
- Training levers: (1) SFT examples that model 'I don't know' correctly (harder than it looks — must not over-refuse); (2) preference data that rewards refusing over confabulating on unknown facts; (3) calibration losses that align the model's stated confidence with observed accuracy; (4) tool use (retrieval, code execution) to verify before answering.
How do you balance helpfulness vs harmlessness in RLHF?
hard- (1) Multi-headed RM: train two reward heads — one for helpfulness, one for harmlessness — and use a weighted combination during PPO.
- Weights explicitly encode the trade-off.
- (2) Constitutional AI approach: separate 'harmless' and 'helpful' phases of preference generation.
- (3) Rejection sampling with a safety filter — sample many completions, filter unsafe ones, then pick the most helpful.
- Anthropic's HH-RLHF paper (Bai 2022) showed multi-head RM is more stable than a single-head one.
Why is PPO used in RLHF instead of vanilla REINFORCE?
hard- REINFORCE has huge gradient variance in the LLM regime (long token sequences, sparse rewards) and requires many samples to reduce it.
- PPO uses a clipped surrogate objective that bounds how far the policy can move per update — much more stable.
- It also uses generalized advantage estimation (GAE) to reduce variance further.
- Trade-off: PPO adds hyperparameters (clip ratio, GAE lambda, value function).
- Modern DPO / GRPO variants further simplify by removing the value function entirely.
What is GRPO?
hard- Group Relative Policy Optimization (DeepSeek 2024).
- Skips the value function of PPO.
- For each prompt, sample K completions, compute their rewards, and use their z-scored advantage relative to the group mean as the PPO gradient signal.
- Advantages: no value network to train (halves memory), simpler, works well for math/code where rewards are ± and rankings within a group are meaningful.
- Used to train DeepSeek-Math and DeepSeek-R1 (reasoning-focused models).
What is iterative DPO / online DPO?
hard- Standard DPO uses a fixed set of preference pairs.
- Iterative DPO: after DPO fine-tuning, generate new completions from the updated model, get preferences (from a judge LLM or humans), and DPO-fine-tune again.
- Repeat.
- Each round moves the reference model forward.
- Improves quality closer to full RLHF at a fraction of the compute.
- Zephyr-Beta, Tulu-3, and Llama-3-Instruct use iterative preference optimization variants.
What is an instruction hierarchy in modern LLMs?
hard- OpenAI (2024) formalized: developer instructions > user instructions > tool outputs > third-party content.
- RLHF training teaches the model to weight higher tiers over lower ones when they conflict.
- Primary defense against indirect prompt injection (retrieved content trying to override the system prompt).
- Anthropic uses a similar 'privileged / unprivileged' distinction.
- Enforced during alignment via curated conflict examples.
How well-calibrated are LLM confidences and how do you fix them?
hard- Base LLMs are decently calibrated (token probabilities match observed accuracy in-distribution).
- RLHF-tuned LLMs are overconfident — asked 'how sure are you?' they say 90% but are right 60% of the time.
- Fixes: (1) verbalized confidence prompts + calibration on a labeled dev set; (2) sampling-based confidence (fraction of K samples that agree); (3) explicit calibration training (e.g., True/False + Confidence-tuning); (4) ensembling.
- Calibration matters for downstream decisions: threshold on the LLM's confidence before deciding to escalate to a human.