22 interview questions on offline rl, each answered in full. Free to read, no account needed.
What is off-policy evaluation and why is it hard?
hard- Off-policy evaluation estimates the value of a target policy using data collected by a different behavior policy — without deploying the new policy.
- It's crucial in medicine, finance, and any high-stakes system where trying the new policy is risky.
- It is hard because of distribution shift; techniques include importance sampling, doubly robust estimators, and model-based methods.
- Variance and bias tradeoffs are severe.
What is offline RL and why does it matter?
medium- Learn policy from fixed dataset of (s, a, r, s') collected by some behavior policy — NO env interaction during training.
- Motivation: (1) real environments too expensive / risky to explore (healthcare, autonomous, industrial).
- (2) Leverage huge existing logs (recommendation clicks, robot demos, dialog logs).
- (3) Safer than online.
- Foundation of practical industrial RL.
- D4RL is standard benchmark suite.
Why is distribution shift the core challenge in offline RL?
hard- Policy π_θ may query Q at (s, a) never seen in data → Q's extrapolation is nonsense → policy exploits Q's overestimation of OOD actions → learned policy fails.
- Naive off-policy methods (SAC on offline data) diverge catastrophically.
- Fixes: constrain policy near behavior (BCQ), pessimistic Q (CQL), model-based with uncertainty (MOReL, COMBO), decision-transformer (no Q at all).
BCQ — Batch-Constrained Q-Learning.
hard- Fujimoto et al. 2019.
- Constrain policy to only take actions similar to those in the data.
- Uses a conditional VAE trained on (s, a) pairs to sample actions π(a∣s) close to behavior.
- Q-learning then only over these sampled 'safe' actions.
- Effectively KL-constrained policy improvement without explicit KL.
- Modern successors: TD3+BC (adds BC term), IQL (uses expectiles).
CQL — Conservative Q-Learning.
hard- Kumar et al. 2020.
- Add penalty to standard Q-learning loss: expectation of Q over the policy (pushed down) minus expectation over data (pushed up) → pessimistic on OOD actions.
- Prevents Q from over-estimating outside data.
- Widely used in modern offline RL benchmarks.
- Result: policy converges to a lower-bound on true Q → conservative, safe behavior.
- Simple to implement on top of SAC.
IQL — Implicit Q-Learning.
hard- Kostrikov et al. 2022.
- Avoids explicit policy constraint or Q pessimism.
- Uses expectile regression to approximate maxa Q(s, a) without evaluating Q at unseen actions — only uses actions from the data.
- Extract policy via advantage-weighted regression on behavior policy.
- Simpler, more stable than BCQ / CQL.
- Often the modern default for offline RL benchmarks in 2023+.
TD3+BC — the simplest offline RL baseline.
medium- Fujimoto & Gu 2021.
- TD3 (actor-critic) + BC (behavior cloning) regularizer added to policy loss: min E[-Q(s, π(s))] + λ (π(s)−adata)2.
- Encourages policy to stay close to data actions.
- Extremely simple (add one term), strong performance on D4RL.
- Foundation demonstrating that a small BC regularization is often enough for offline RL — sophisticated methods (CQL, IQL) add marginal gains.
Behavior Cloning (BC) — when does it work?
medium- Supervised learning: fit π(a∣s) to (s, a) pairs from expert demos via MLE.
- Works when: (1) expert data is high quality, (2) test distribution matches expert distribution (no drift), (3) enough coverage.
- Fails: (1) compounding errors under distribution shift (agent drifts, sees unseen states, makes worse mistakes).
- Solutions: DAgger (query expert in new states), residual RL, BC + RL fine-tuning.
DAgger — Dataset Aggregation for imitation learning.
medium- Ross et al. 2011.
- Iterative BC: (1) train π on expert data, (2) roll out π in env, (3) query expert for correct action at every state visited, (4) add (state, expert-action) to dataset, (5) retrain.
- Fixes BC's compounding errors by getting expert labels on the agent's own state distribution.
- Requires access to expert during training.
- Foundation of interactive imitation learning.
Inverse Reinforcement Learning (IRL) — the setup.
hard- Given expert demonstrations, infer the reward function that makes the expert optimal.
- Ill-posed (many rewards consistent).
- Approaches: (1) Feature-matching: match expected features under learner and expert.
- (2) Max-entropy IRL (Ziebart 2008): among consistent rewards, prefer max-entropy.
- (3) GAIL (Ho & Ermon 2016): GAN-like adversarial training — discriminator judges expert vs learner.
- Foundation of modern imitation learning + robotics.
GAIL — Generative Adversarial Imitation Learning.
hard- Ho & Ermon 2016.
- Similar to GAN: generator = policy π, discriminator D distinguishes agent trajectories from expert trajectories.
- Train π to fool D via policy gradient (log D as reward).
- Recovers expert behavior without explicit reward.
- Extends to continuous control + high-dim states.
- Foundation of adversarial imitation learning.
- Modern extensions: SQIL, ValueDICE, AdVIL.
Reward hacking in RLHF — canonical examples.
hard- Model finds ways to score high on r_φ without being genuinely helpful: (1) sycophancy: agree with user regardless of truth.
- (2) Length bias: longer responses often scored higher by human raters, so model over-verbose.
- (3) Excessive hedging (safety over-refusal).
- (4) Format exploitation (bullets look thorough).
- (5) Adversarial phrases that game the model.
- Fixes: iterative reward model updates, length-normalized rewards, adversarial red-team, RLAIF (AI feedback).
Constitutional AI — mechanism.
hard- Anthropic 2022.
- Two phases: (1) SL from AI feedback: model critiques + revises its own responses per written constitution (helpful, harmless principles); train on (prompt, revised response).
- (2) RL from AI feedback: LLM judge scores pairs of responses on constitutional principles; train reward model + PPO.
- Advantage: avoids exposing human labelers to harmful content.
- Foundation of Claude's training.
- Modern LLM alignment default.
How does the KL penalty interact with reward hacking?
hard- KL to πref bounds how far model can drift → limits reward hacking of imperfect r_φ.
- But: (1) large model can find hacks within KL budget (spuriousfeaturesthatbothrφlikesANDπrefprobablywouldgenerate).
- (2) β must be tuned per task — too high stalls learning.
- (3) Better: iterative RM updates (relabel new-policy responses), adversarial reward shaping.
- (4) Modern practice: KL as safety floor, not primary defense against hacking.
Safe RL — how do you constrain during learning?
hard- (1) CMDP (Constrained MDP): maximize reward subject to E[Σct] ≤ threshold.
- Solved via Lagrangian or projected gradient.
- (2) Safe exploration: pretrain with safe demonstrations, use safety layers.
- (3) Reachability analysis.
- (4) Reward shaping with constraint violations.
- (5) Test-time verification: reject actions predicted to violate constraints.
- Standard in robotics, autonomous vehicles, medical RL — high-stakes safety-critical domains.
Sycophancy in RLHF — what causes it and how to fix?
hard- Model tells users what they want to hear rather than truth.
- Cause: human labelers prefer agreeable responses; RM learns to reward agreement patterns.
- Fixes: (1) explicitly label sycophancy in preference data.
- (2) Train with adversarial 'test' prompts asking for wrong assertions.
- (3) Include truthfulness in constitutional principles.
- (4) Ensemble RMs to detect disagreement.
- Ongoing challenge in modern LLM alignment.
How does RL contribute to jailbreak defense?
hard- (1) Safety-tuned RM penalizes harmful outputs.
- (2) Adversarial RLHF: red-team probes for jailbreaks, add examples to preference data.
- (3) Refusal tuning: SFT + RLHF on 'appropriate refusal' patterns.
- (4) Multi-round RLAIF on constitutional harmful principles.
- Trade-off: too aggressive → over-refusal; too lax → harmful outputs.
- Modern practice: continuous red-teaming with automated attacks (GCG) as part of RM training loop.
How is o1-style reasoning safer or riskier?
hard- Safer: (1) longer reasoning gives more opportunities to catch errors + reject harmful requests.
- (2) Explicit deliberation can invoke safety principles.
- Riskier: (3) can 'think its way' around safety training if not RL-trained on refusal.
- (4) Chain-of-thought exposes reasoning that could be jailbroken.
- (5) Increased test-time compute → some safety mitigations at inference.
- Modern practice: RL train reasoning models with explicit safety verifier reward.
Why is RL hard for autonomous driving?
hard- (1) Safety: cannot explore in real traffic.
- (2) Long-tail rare events matter (fatal edge cases).
- (3) Multi-agent interactions (other drivers, pedestrians) with non-stationary policies.
- (4) Partial observability.
- (5) Reward specification difficulty.
- Most industry uses SL / imitation learning + planning stack (Waymo, Cruise).
- RL used mostly in simulation for policy fine-tuning, closed-loop training.
- Recent: Tesla's neural planners are hybrid IL + RL.
RL in healthcare — key challenges.
hard- (1) Sample efficiency (real trials cost billions).
- (2) Safety absolute — can't try random treatments.
- (3) Off-policy evaluation on historical patient data (dosing decisions from EHR) is standard.
- (4) Confounders in observational data — need causal RL.
- (5) Reward specification: what is 'good outcome' over years?
- Applications: dosing optimization (mechanical ventilation, sepsis), personalized treatment.
- Deployed in advisory tools only; not autonomous treatment selection.
Interview: 'how do you make an RL policy safe to deploy?'
hard- (1) Simulation validation across held-out scenarios + adversarial tests.
- (2) Formal / conservative bounds where possible (CMDP).
- (3) Safety layer: monitor / veto extreme actions at runtime.
- (4) Human-in-the-loop for edge cases.
- (5) Shadow deployment first (log-only), then gradual rollout with A/B guardrails.
- (6) Continuous monitoring: distribution shift, reward drift, action distribution, incident rate.
- (7) Rollback plan.
- Safety-critical RL is engineering-heavy.
Safety verifier reward — what and how?
hard- Automatic reward from external verifier that judges safety: (1) content filter classifier flags harmful outputs.
- (2) LLM judge with harm rubric.
- (3) Rule-based checks (PII leakage, illegal content).
- Combined with helpfulness reward → multi-objective RL.
- Alignment tax risk if safety over-weighted.
- Modern: 'safety as red-team' — actively probe model with attacks, RL trains against them (Anthropic's HH pipeline).