EasyDeepLearn

All 214 Reinforcement Learning interview questions

MDPs, exploration, policy vs value methods, and modern actor-critic algorithms.

Every question below opens on its own page, with a full answer, the formulas written out and a multiple-choice check. Free to read, no account needed.

LessonFlashcardsQuizMock

Fundamentals & MDPs

  1. Define the RL setting: agent, environment, state, action, reward, policy.easy
  2. On-policy vs off-policy learning — what's the difference?medium
  3. What does the Bellman equation say?medium
  4. What role does the discount factor gamma play?easy
  5. What is the credit assignment problem?hard
  6. Model-based vs model-free RL — when do you prefer each?medium
  7. What is the Markov property and why does it matter?medium
  8. POMDP — how does it differ from MDP?hard
  9. V(s) vs Q(s, a) — the difference.easy
  10. What is the advantage function?medium
  11. Different notions of return — MC, TD(0), n-step, GAE.hard
  12. What is TD error?medium
  13. Policy iteration vs value iteration.medium
  14. TD(λ) — how does it work?hard
  15. Monte Carlo methods in RL — when to use?medium
  16. Tabular vs function-approximation RL — the challenge.hard
  17. Why do Bellman operators converge?hard
  18. TD vs MC — how they differ in bias-variance.medium
  19. Linear function approximation for value functions — pros and cons.medium
  20. When does tabular Q-learning provably converge?hard
  21. When does tabular Q-learning fail in practice?medium
  22. Why is subtracting a baseline OK in policy gradient?hard
  23. The log-derivative trick — why is it central to policy gradients?hard
  24. What is natural gradient in policy gradient methods?hard
  25. Reparameterization trick in stochastic policies.hard
  26. GAE — Generalized Advantage Estimation formula.hard
  27. Off-policy policy gradient — how does importance sampling enter?hard
  28. V-trace off-policy correction formula.hard
  29. Retrace(λ) — safe off-policy Q updates.hard
  30. Policy gradient as mirror descent — the connection.hard
  31. UCB — Upper Confidence Bound for exploration.medium
  32. Model bias in MBRL — the challenge.hard
  33. LQR — Linear Quadratic Regulator and where it's still used.medium
  34. When does planning beat learning?medium
  35. Decision Transformer — how does it recast RL?hard
  36. Trajectory Transformer — variant of Decision Transformer.hard
  37. Nash equilibrium in multi-agent RL — what and when?hard
  38. Fictitious play — what is it?hard
  39. Options framework — formalization.hard
  40. DPO derivation — the key mathematical insight.hard
  41. Meta-RL — learning to learn.hard
  42. How does transfer learning work in RL?medium
  43. Successor features — what and why?hard
  44. Reward machines — structured reward specification.hard
  45. Potential-based reward shaping — why is it safe?hard
  46. How do you handle partial observability in deep RL?medium
  47. LQG — Linear Quadratic Gaussian control.hard
  48. Generalization in RL — why is it hard?hard
  49. Catastrophic forgetting in continual RL.hard
  50. Generalist agents — Gato and DeepMind's approach.hard
  51. Quantile regression in QR-DQN — the loss.hard
  52. Deadly triad — the three ingredients.hard
  53. Where is RL heading (2025+)?hard
  54. How do you choose the discount factor, and what goes wrong at the extremes?medium
  55. What is the deadly triad and how do practical algorithms cope with it?hard
  56. Why can't you just run Q-learning on a fixed dataset?hard

Value methods (Q-learning, DQN)

  1. What update does Q-learning perform?medium
  2. What are the key tricks that make DQN work?hard
  3. Double DQN — what problem does it solve?medium
  4. Dueling DQN — architecture.hard
  5. Prioritized replay — mechanism.hard
  6. Rainbow DQN — what six extensions does it combine?hard
  7. Distributional RL (C51, QR-DQN) — the idea.hard
  8. Noisy nets — how do they replace ε-greedy?hard
  9. SARSA update — on-policy analog of Q-learning.medium
  10. How do you size a replay buffer?medium
  11. Hard update vs soft (Polyak) update for target networks.medium
  12. Why does DQN scale poorly to continuous action spaces?hard
  13. Fitted Q-Iteration — batch off-policy Q-learning.hard
  14. C51 — how does the projected Bellman update work?hard
  15. IQN — Implicit Quantile Networks.hard
  16. Why is naive DQN unstable and what specific fixes address which cause?hard
  17. Value decomposition (VDN, QMIX) — for multi-agent Q-learning.hard
  18. DDPG — Deep Deterministic Policy Gradient.hard
  19. TD3 — Twin Delayed DDPG improvements over DDPG.hard
  20. SAC — Soft Actor-Critic mechanism and advantages.hard
  21. IMPALA — how does it scale actor-critic?hard
  22. Vectorized environments — why and how?medium
  23. Bootstrapped DQN — approximate Thompson sampling for deep RL.hard
  24. Count-based exploration for large state spaces.hard
  25. Random Network Distillation (RND) — mechanism.hard
  26. Intrinsic Curiosity Module (ICM) — Pathak et al.hard
  27. Go-Explore — how does it solve hard-exploration?hard
  28. Hindsight Experience Replay (HER) — the trick.hard
  29. Agent57 — how did it achieve above-human on all Atari?hard
  30. World models (Ha & Schmidhuber, Dreamer) — big idea.hard
  31. MuZero — model-based RL without knowing the env dynamics.hard
  32. MBPO — Model-Based Policy Optimization.hard
  33. How do you improve sample efficiency in deep RL?medium
  34. Sim-to-real — main techniques.hard
  35. R2D2 — distributed recurrent DQN.hard
  36. Why does DQN stack 4 consecutive frames?easy
  37. Reward clipping in DQN — trade-offs.medium
  38. OpenAI Five — Dota 2 milestone.hard
  39. Plasticity loss in deep RL — the problem.hard

Policy gradients & actor-critic

  1. What is a policy gradient and why do we use it?hard
  2. What is an actor-critic algorithm?medium
  3. Why is PPO so widely used?medium
  4. REINFORCE — the vanilla policy-gradient algorithm.medium
  5. TRPO — Trust Region Policy Optimization.hard
  6. PPO's clipped surrogate objective — write it and explain.hard
  7. PPO implementation details that actually matter.hard
  8. A2C vs A3C — the difference.medium
  9. Why add an entropy bonus to the policy objective?medium
  10. PPO vs SAC — which do you pick?medium
  11. DPO — Direct Preference Optimization instead of RLHF PPO.hard
  12. GRPO — Group Relative Policy Optimization (DeepSeekMath, R1).hard
  13. Categorical vs Gaussian vs beta policies — which and when?medium
  14. Value function clipping in PPO — why and how?hard
  15. How does batch size / rollout length affect PPO?medium
  16. Ornstein-Uhlenbeck noise vs Gaussian noise for continuous control.medium
  17. MADDPG — Multi-Agent DDPG.hard
  18. MAPPO — Multi-Agent PPO.medium
  19. Hierarchical RL — why and how?hard
  20. DIAYN — Diversity is All You Need for skill discovery.hard
  21. Goal-conditioned RL — setup.medium
  22. Interview: 'explain why PPO is on-policy but uses a ratio to importance-sample.'hard
  23. Curriculum learning in RL — how to design one?medium
  24. What is the clipping in PPO actually protecting you from?hard
  25. When does the sample inefficiency of on-policy methods stop mattering?medium

Exploration strategies

  1. What is the exploration-exploitation tradeoff?easy
  2. CEM — Cross-Entropy Method for planning.medium
  3. ε-greedy — how do you schedule ε?easy
  4. How does posterior sampling drive exploration in deep RL?hard
  5. Why is Montezuma's Revenge such a famous benchmark?medium
  6. Parameter noise for exploration — how is it different?medium
  7. The 'noisy TV problem' in curiosity-driven exploration.hard
  8. Emergent tool use in RL — canonical example.medium
  9. You want to optimize which of five banners to show. Bandit or full RL?medium
  10. How do you explore safely when the environment is real paying users?hard
  11. The reward arrives only at the end of a 500-step episode. What do you do?hard

Model-based & planning

  1. Monte Carlo Tree Search — how it works in AlphaGo/MuZero.hard
  2. AlphaZero — key ideas.hard
  3. Dyna — early model-based RL framework.medium
  4. Model Predictive Control (MPC) in RL — how does it work?medium

Reward design & shaping

  1. What is reward shaping and what are its pitfalls?medium
  2. Reward engineering — practical guidelines.medium
  3. How do you catch reward hacking before it reaches production?hard

Deep RL engineering

  1. Learning-rate schedule for PPO / SAC — what works?medium
  2. Population-Based Training (PBT) for RL.hard
  3. Interview: 'your DQN agent isn't learning — how do you debug?'hard
  4. Policy averaging — what is it and why?medium
  5. How do you efficiently train RLHF at scale?hard
  6. Your policy is perfect in simulation and useless on the real robot. How do you close the gap?hard
  7. Two RL algorithms report different scores in a paper. Why should you be skeptical?medium

Offline RL

  1. What is off-policy evaluation and why is it hard?hard
  2. What is offline RL and why does it matter?medium
  3. Why is distribution shift the core challenge in offline RL?hard
  4. BCQ — Batch-Constrained Q-Learning.hard
  5. CQL — Conservative Q-Learning.hard
  6. IQL — Implicit Q-Learning.hard
  7. TD3+BC — the simplest offline RL baseline.medium
  8. Behavior Cloning (BC) — when does it work?medium
  9. DAgger — Dataset Aggregation for imitation learning.medium
  10. Inverse Reinforcement Learning (IRL) — the setup.hard
  11. GAIL — Generative Adversarial Imitation Learning.hard
  12. Reward hacking in RLHF — canonical examples.hard
  13. Constitutional AI — mechanism.hard
  14. How does the KL penalty interact with reward hacking?hard
  15. Safe RL — how do you constrain during learning?hard
  16. Sycophancy in RLHF — what causes it and how to fix?hard
  17. How does RL contribute to jailbreak defense?hard
  18. How is o1-style reasoning safer or riskier?hard
  19. Why is RL hard for autonomous driving?hard
  20. RL in healthcare — key challenges.hard
  21. Interview: 'how do you make an RL policy safe to deploy?'hard
  22. Safety verifier reward — what and how?hard

Multi-agent RL

  1. Multi-agent RL — what makes it fundamentally different?hard
  2. Centralized Training Decentralized Execution (CTDE).hard
  3. Self-play — why does it work for games?hard
  4. League play in AlphaStar — mechanism.hard
  5. How do agents learn to cooperate?hard
  6. Emergent communication in MARL — canonical result.hard
  7. AlphaStar StarCraft II — main technical innovations.hard

RLHF & LLM alignment

  1. What is RLHF and how does it fit into training an LLM?hard
  2. RLHF pipeline — the three stages in detail.hard
  3. Bradley-Terry model in reward modeling.hard
  4. Why does RLHF include a KL penalty?hard
  5. DPO vs PPO for RLHF — practical comparison.hard
  6. RLAIF — RL from AI Feedback.hard
  7. RL for reasoning (o1, DeepSeek-R1) — the paradigm shift.hard
  8. How does RL train chain-of-thought reasoning?hard
  9. Verifier reward vs preference-based reward — when to use each?medium
  10. Process reward vs outcome reward for reasoning.hard
  11. Iterative RLHF — why do you keep going?hard
  12. What can RL do that SFT cannot?medium
  13. How do reward models scale with size?hard
  14. How do you fix length bias in RLHF?medium
  15. ORPO — Odds Ratio Preference Optimization.hard
  16. KTO — Kahneman-Tversky Optimization.hard
  17. What is 'alignment tax' in RLHF?hard
  18. How does RL train tool-use in LLM agents?hard
  19. Interview: 'walk me through the tradeoffs of SFT-only vs SFT+RLHF vs SFT+DPO.'hard
  20. ORPO vs DPO — key difference.hard
  21. LLM agent benchmarks — what tests capability?medium
  22. 'Verifier + refiner' RL pattern for reasoning.hard
  23. In RLHF, why does the reward model become unreliable as training progresses?hard
  24. Why do many teams choose DPO over PPO for alignment?medium

Applications & safety

  1. AlphaGo — what made it different from prior Go engines?medium
  2. Is AlphaFold RL?hard
  3. OpenAI's Rubik's Cube robot — key techniques.hard
  4. Modern quadruped locomotion — RL pipeline.hard
  5. Industrial RL applications and challenges.medium
  6. How is a recommender system a bandit / RL problem?medium
  7. RL in trading / finance — techniques.hard
  8. Interview: 'design exploration for a recommendation system.'hard
  9. Standard deep RL benchmark suites.medium
  10. RT-2 — Vision-Language-Action model.hard
  11. A product manager asks for reinforcement learning. When should you talk them out of it?medium

Interview scenarios

  1. Interview: 'design an RL system for X' — what's your framework?medium
  2. Interview: 'when should you NOT use RL?'medium
  3. Interview: 'you're given 100 hours of expert data + can simulate — how do you train an agent?'hard
  4. Interview: 'DQN vs PPO vs SAC vs DPO — which for what?'medium
  5. Interview: 'summarize when RL wins in production.'medium

Other topics