56 interview questions on fundamentals & mdps, each answered in full. Free to read, no account needed.
Define the RL setting: agent, environment, state, action, reward, policy. easy An agent observes a state s from the environment, picks an action a via its policy π ( a ∣ s ) \pi(a \mid s) π ( a ∣ s ) The goal is to maximize the expected cumulative discounted reward (return). Formalized as a Markov Decision Process (S, A, P, R, gamma). The agent learns pi (or a value function) from interaction with the environment.
On-policy vs off-policy learning — what's the difference? medium On-policy methods learn from data generated by the current policy (e.g., SARSA, REINFORCE, PPO). Off-policy methods learn from data collected by a different (behavior) policy — this enables replay buffers and reusing old data (e.g., Q-learning, DQN, SAC, DDPG). Off-policy is more sample-efficient; on-policy is often more stable and easier to tune.
What does the Bellman equation say? medium For a policy pi: V π ( s ) = E [ r + γ ⋅ V π ( s ) ] V_{\pi}(s)\; = \;E[r\; + \;\gamma\; \cdot \;V_{\pi}(s)] V π ( s ) = E [ r + γ ⋅ V π ( s )] For the optimal value: V ⋅ ( s ) = max a V \cdot (s)\; = \;\operatorname{max}_{a} V ⋅ ( s ) = max a It expresses the value of a state as the immediate reward plus the discounted value of the next state. Solving the Bellman equation is the core of dynamic programming, TD learning, and Q-learning.
What role does the discount factor gamma play? easy Gamma in [0, 1) weights how much future rewards matter compared to immediate ones. Gamma close to 0 = myopic (only care about immediate reward); gamma close to 1 = far-sighted (care about long-term). Also ensures the return sum converges in infinite-horizon problems. Choose based on the task horizon and stability of value estimates.
What is the credit assignment problem? hard In sequential tasks with delayed rewards, it's hard to know which past action caused a given reward. Long horizons, sparse rewards, and stochasticity make it worse. Methods that help: temporal-difference learning, eligibility traces, n-step returns, GAE (generalized advantage estimation), and hindsight experience replay.
Model-based vs model-free RL — when do you prefer each? medium Model-based RL learns a model of the environment (transitions and rewards) and uses it to plan (e.g., MuZero, Dreamer). It is more sample-efficient — great when real interactions are expensive (robotics, healthcare). Model-free RL learns policy or values directly from experience — simpler, and it wins when the environment is complex/hard to model. Modern successes often combine both.
What is the Markov property and why does it matter? medium P ( s t + 1 ∣ s t , a t , s t − 1 , , s 0 ) = P ( s t + 1 ∣ s t , a t ) P(s_{t + 1}\; \mid \;s_{t}, \;a_{t}, \;s_{t - 1}, \;, \;s_{0})\; = \;P(s_{t + 1}\; \mid \;s_{t}, \;a_{t}) P ( s t + 1 ∣ s t , a t , s t − 1 , , s 0 ) = P ( s t + 1 ∣ s t , a t ) Enables efficient RL: policy and value functions depend on state alone, not history. When violated (partial observability), we use POMDPs, recurrent nets, or state-stacking (Atari frame stack) to recover a Markovian representation.
POMDP — how does it differ from MDP? hard Partially Observable MDP: agent observes o t o_{t} o t s t s_{t} s t Add observation function O ( o ∣ s ) O(o\; \mid \;s) O ( o ∣ s ) Belief state b(s) = posterior over s given history → sufficient statistic for optimal policy. Solving POMDPs is PSPACE-hard exactly; approximate via RNN/LSTM encoding of history, or particle filter beliefs, or transformer over recent observations. Realistic robotics / partial-info games.
V(s) vs Q(s, a) — the difference. easy V(s) = expected return from state s following policy π. Q(s, a) = expected return from state s taking action a THEN following π. Relationship: V ( s ) = Σ a V(s)\; = \;{\Sigma}_{a} V ( s ) = Σ a π ( a ∣ s ) {\pi}(a \mid s) π ( a ∣ s ) Optimal: V ⋅ ( s ) = max a V \cdot (s)\; = \;\operatorname{max}_{a} V ⋅ ( s ) = max a Q allows action selection without a model (Q-learning); V requires a model to compute argmax over actions. Actor-critic methods often estimate V (simpler) but use per-action advantage A = Q - V.
What is the advantage function? medium A(s, a) = Q(s, a) - V(s) → how much better is action a than the average action in state s. Reduces variance in policy-gradient estimates (baseline subtraction preserves the gradient in expectation but shrinks variance). Standard estimators: TD-residual A ≈ r + γ V(s') - V(s) (biased low variance); n-step or GAE for bias-variance tradeoff. Used in A2C, PPO, TRPO.
Different notions of return — MC, TD(0), n-step, GAE. hard MC return: G t = Σ G_{t}\; = \;{\Sigma} G t = Σ γ k {\gamma}^{k} γ k r t + k r_{t + k} r t + k TD(0): r t + γ r_{t}\; + \;{\gamma} r t + γ V ( s t + 1 ) V(s_{t + 1}) V ( s t + 1 ) r t + γ r_{t}\; + \;{\gamma} r t + γ r t + 1 r_{t + 1} r t + 1 γ n − 1 {\gamma}^{n - 1} γ n − 1 r t + n − 1 + γ n r_{t + n - 1}\; + \;{\gamma}^{n} r t + n − 1 + γ n V ( s t + n ) V(s_{t + n}) V ( s t + n ) GAE (Generalized Advantage Estimation): weighted sum over n-step advantages with parameter λ — smooth bias-variance tradeoff. GAE with λ ≈ 0.95 is default in PPO.
What is TD error? medium δ t = r t + γ {\delta}_{t}\; = \;r_{t}\; + \;{\gamma} δ t = r t + γ V ( s t + 1 ) − V ( s t ) V(s_{t + 1})\; - \;V(s_{t}) V ( s t + 1 ) − V ( s t ) Drives all TD-based updates: V ← V + α δ, Q-learning target is δ with max in bootstrap. Positive δ = state was better than expected; negative = worse. Fundamental signal for value learning. Modern uses: PPO advantage, GAE, prioritized replay weighting (δ magnitude).
Policy iteration vs value iteration. medium Policy iteration: alternate (a) policy evaluation ( c o m p u t e V π b y s o l v i n g B e l l m a n u n t i l c o n v e r g e n c e ) (\mathrm{compute}\;V{\pi}\;\mathrm{by}\;\mathrm{solving}\;\mathrm{Bellman}\;\mathrm{until}\;\mathrm{convergence}) ( compute V π by solving Bellman until convergence ) ( π ← g r e e d y o n V π ) ({\pi}\; \leftarrow \;\mathrm{greedy}\;\mathrm{on}\;V{\pi}) ( π ← greedy on V π ) Converges in finite steps. Value iteration: V ← max a \operatorname{max}_{a} max a Value iteration is a special case of policy iteration with only one evaluation sweep. Both are dynamic programming, require the model.
TD(λ) — how does it work? hard Interpolates between TD(0) (λ=0, low variance biased) and MC (λ=1, unbiased high variance). Uses eligibility traces e t ( s ) e_{t}(s) e t ( s ) Update: V(s) ← V(s) + α δ e(s) for every s. Online, single-pass, works with function approximation. Foundation of Sutton-Barto RL; GAE is essentially TD(λ) for advantages.
Monte Carlo methods in RL — when to use? medium Estimate V_π(s) or Q_π(s, a) via sample-average returns from full episodes. Unbiased (no bootstrap error), but only usable in episodic tasks + high variance. Every-visit MC: average over every visit to s in an episode. First-visit MC: only first occurrence. Rare in modern deep RL (TD methods better for long / continuous tasks) but foundation of return-conditioned methods and offline RL (Decision Transformer).
Tabular vs function-approximation RL — the challenge. hard Tabular: one entry per (s, a) — convergence guaranteed but doesn't scale. Function approximation (neural nets): scales to huge state spaces but breaks convergence guarantees. The 'deadly triad': off-policy + function approximation + bootstrapping → potential divergence. Modern deep RL uses tricks (target networks, replay, gradient clipping) to work despite the triad. Understanding this trap is key.
Why do Bellman operators converge? hard Bellman operator T is a γ-contraction in the max norm: ||T V 1 − T V 2 \mathrm{TV}_{1}\; - \;\mathrm{TV}_{2} TV 1 − TV 2 V 1 − V 2 V_{1}\; - \;V_{2} V 1 − V 2 By Banach fixed-point theorem, iterating T converges to unique fixed point V* at rate γ k {\gamma}^{k} γ k Underlies value iteration, TD(0) convergence in tabular case, policy iteration. Function approximation can break the contraction (projected Bellman operator may not contract) → divergence.
TD vs MC — how they differ in bias-variance. medium MC: use full episodic return G t G_{t} G t TD(0): use r t + γ r_{t}\; + \;{\gamma} r t + γ Choice: TD when episodes are long or variance dominates, MC when bias is intolerable + episodes short.
Linear function approximation for value functions — pros and cons. medium V(s) ≈ w'φ(s) with hand-crafted features φ. Convergence guarantees under on-policy (LSPI, LSTD) but poor scaling; requires domain-designed features. Baird's counterexample: off-policy + linear FA + bootstrap → divergence. Foundation of classic RL theory (Sutton, Bertsekas), superseded by neural nets in practice. Still valuable in tiny state spaces, small-embedded systems, or theoretical proofs.
When does tabular Q-learning provably converge? hard (1) All (s, a) visited infinitely often (adequate exploration), (2) Learning rate α satisfies Robbins-Monro: Σα = ∞, Σ α 2 {\Sigma}{\alpha}^{2} Σ α 2 Under these, Q k Q_{k} Q k Real deep-RL breaks (1) severely (state space huge) and mostly ignores (2). Classical theory guides intuition but doesn't guarantee anything in modern deep RL — that's why empirical care matters.
When does tabular Q-learning fail in practice? medium (1) State space too large: even discretized MazeGrid > 1M states makes tabular infeasible. (2) Continuous features: discretization fights curse of dimensionality. (3) Function structure across states unexploited (Q for nearby states should be similar → NN generalizes; tabular treats them independently). (4) Slow: no generalization means each state must be visited separately. Every real RL problem > toy uses function approximation (usually NN).
Why is subtracting a baseline OK in policy gradient? hard Because E [ b ( s ) ∇ log π ( a ∣ s ) ] = 0 E[b(s)\; \nabla \operatorname{log}\;{\pi}(a \mid s)]\; = \;0 E [ b ( s ) ∇ log π ( a ∣ s )] = 0 This adds no bias but shrinks variance by centering the reward signal. Choice of baseline: constant, moving average of returns, or state-dependent V(s) (best — cancels state-level variance). Advantage function A(s, a) = Q(s, a) - V(s) is the popular choice → gives us actor-critic architectures.
The log-derivative trick — why is it central to policy gradients? hard For expectation of f under π_θ: ∇_θ E_π[f] = E_π[ f ∇ θ log π θ ] [f\; \nabla {\theta}\;\operatorname{log}\;{\pi}{\theta}] [ f ∇ θ log π θ ] Enables Monte Carlo estimation of gradient using only samples from π and gradient of log-likelihood — no need to differentiate through the reward function or the environment. Foundation of REINFORCE, actor-critic, natural gradient, and importance sampling. Trick: π_θ(x) ∇_θ log π_θ(x) = ∇_θ π_θ(x).
What is natural gradient in policy gradient methods? hard Follow the direction of steepest ascent in policy space measured by KL divergence (Fisher information metric) instead of Euclidean. Update: θ ← θ + α F ( θ ) − 1 F({\theta}) - 1 F ( θ ) − 1 Invariant to reparameterization. Foundation of TRPO (approximate) and NPG. Practically: F − 1 F^{- 1} F − 1 Underpins why PPO's clipping approximates a trust region.
Reparameterization trick in stochastic policies. hard For continuous stochastic policy π_θ( a ∣ s ) = N ( μ θ ( s ) , σ θ ( s ) 2 ) (a \mid s)\; = \;N({\mu}{\theta}(s), \;{\sigma}{\theta}(s)^{2}) ( a ∣ s ) = N ( μ θ ( s ) , σ θ ( s ) 2 ) Enables low-variance path-wise gradients rather than score-function REINFORCE-style gradients. Used in SAC. Also foundation of VAEs. Doesn't work directly for discrete actions (use Gumbel-softmax as continuous relaxation).
GAE — Generalized Advantage Estimation formula. hard A t G A E ( γ , λ ) = Σ l = 0 A_{t}^{\mathrm{GAE}({\gamma}, \;{\lambda})}\; = \;{\Sigma}_{l = 0} A t GAE ( γ , λ ) = Σ l = 0 ( γ λ ) l ({\gamma}{\lambda})l ( γ λ ) l δ t + l {\delta}_{t + l} δ t + l δ t = r t + γ {\delta}_{t}\; = \;r_{t}\; + \;{\gamma} δ t = r t + γ V ( s t + 1 ) − V ( s t ) V(s_{t + 1})\; - \;V(s_{t}) V ( s t + 1 ) − V ( s t ) Typical λ = 0.95 in PPO. Provides smooth bias-variance tradeoff. Compute efficiently backward from end of trajectory. Standard advantage estimator in modern policy gradient methods.
Off-policy policy gradient — how does importance sampling enter? hard Data collected under behavior π b {\pi}_{b} π b ( a ∣ s ) / π b ( a ∣ s ) (a \mid s) / {\pi}_{b}(a \mid s) ( a ∣ s ) / π b ( a ∣ s ) Weight variance explodes when π_θ far from π b {\pi}_{b} π b Solutions: clip ratio (PPO), truncate importance weights (Retrace, V-trace in IMPALA), or use deterministic policy gradient (no IS needed — DDPG/SAC).
V-trace off-policy correction formula. hard v s = V ( s s ) + Σ t v_{s}\; = \;V(s_{s})\; + \;{\Sigma}_{t} v s = V ( s s ) + Σ t γ t − s {\gamma}^{t - s} γ t − s ( Π ρ i ) ({\Pi}\;{\rho}_{i}) ( Π ρ i ) δ t V {\delta}_{t}^{V} δ t V δ t V = r t + γ {\delta}_{t}^{V}\; = \;r_{t}\; + \;{\gamma} δ t V = r t + γ V ( s t + 1 ) − V ( s t ) V(s_{t + 1})\; - \;V(s_{t}) V ( s t + 1 ) − V ( s t ) ρ i = min ( ρ , π / μ ) {\rho}_{i}\; = \;\operatorname{min}({\rho}, \;{\pi} / {\mu}) ρ i = min ( ρ , π / μ ) Different clipping thresholds for ρ (target) vs c (trace) allow bias-variance trade. Off-policy corrections without exploding variance. Enables IMPALA-style distributed training. Modern reference for scalable off-policy actor-critic.
Retrace(λ) — safe off-policy Q updates. hard Munos et al. 2016. Q-target: r + γ Σ l {\Sigma}_{l} Σ l ( γ λ ) l ({\gamma}{\lambda})l ( γ λ ) l Π i = 1 l {\Pi}_{i = 1}^{l} Π i = 1 l c i c_{i} c i δ t + l {\delta}_{t + l} δ t + l c i = λ c_{i}\; = \;{\lambda} c i = λ min ( 1 , π / π b ) \operatorname{min}(1, \;{\pi} / {\pi}_{b}) min ( 1 , π / π b ) Guaranteed convergence in off-policy setting + low variance from truncation. Underlies ACER algorithm. Elegant unification of importance sampling + eligibility traces. Modern off-policy actor-critics still cite Retrace's design principles.
Policy gradient as mirror descent — the connection. hard Policy gradient can be viewed as mirror descent on the policy simplex with the KL divergence as Bregman divergence. Natural gradient = mirror descent step in log-policy space. TRPO's KL constraint is explicit mirror-descent step. This unification (Kakade & Langford, later Vieillard et al.) clarifies why entropy regularization, KL constraints, and log-space parameterization all improve stability.
UCB — Upper Confidence Bound for exploration. medium In bandits: pick a = argmax [ Q ( a ) + c ( ln t / n a ) ] [Q(a)\; + \;c\; \sqrt (\operatorname{ln}\;t\; / \;n_{a})] [ Q ( a ) + c ( ln t / n a )] Confidence bound term inflates rarely-tried actions. Provides O(√T log T) regret bound. In tree search (PUCT): a = argmax Q + c · π p r i o r {\pi}_{\mathrm{prior}} π prior N / ( 1 + n a ) N\; / \;(1\; + \;n_{a}) N / ( 1 + n a ) Uses: bandit algorithms, MCTS, contextual bandits. Modern application: exploration bonus in some RL papers (BOOTSTRAPPED DQN, RND).
Model bias in MBRL — the challenge. hard Learned model has errors that compound over rollout → policy trained in imagined trajectories exploits model exploits. Solutions: (1) short rollouts (MBPO uses only 1-15 steps of imagination between real data), (2) ensemble of models for uncertainty (PETS: use variance to penalize plans), (3) careful data collection (COMBO, MOReL) that regularizes toward known regions, (4) uncertainty-aware planning (avoid uncertain trajectories).
LQR — Linear Quadratic Regulator and where it's still used. medium For linear dynamics x t + 1 = A x_{t + 1}\; = \;A x t + 1 = A x t + B x_{t}\; + \;B x t + B u t + n o i s e u_{t}\; + \;\mathrm{noise} u t + noise Solve via discrete-time Riccati equation. Closed-form + fast + optimal for LQ systems. Basis of iLQR (linearize around trajectory + LQR iteratively) — used in robotics locomotion + humanoid control. Modern MBRL uses iLQR / LQR inside planning loops.
When does planning beat learning? medium (1) Compact known dynamics (games, physics). (2) Long horizons where credit assignment is hard for learning. (3) Rare edge cases (planning can reason zero-shot; learning needs data). (4) Test time compute available. AlphaZero + MuZero show planning + learning outperform either alone. Modern LLM reasoning (o1, DeepSeek-R1) is analogous: test-time 'thinking' as planning augments learned priors.
Decision Transformer — how does it recast RL? hard Chen et al. 2021: treat RL as sequence modeling. Given (return-to-go, state, action) sequence, predict next action autoregressively. Trained on offline trajectories via cross-entropy. At inference, condition on desired return-to-go. Fully removes TD learning + bootstrapping — no value functions. Competitive with offline RL on D4RL. Foundation of return-conditioned RL and modern LLM-based agents.
Trajectory Transformer — variant of Decision Transformer. hard Janner et al. 2021: model entire trajectory ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ) (s_{0}, \;a_{0}, \;r_{0}, \;s_{1}, \;a_{1}, \;r_{1}, \;) ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ) Planning via beam search over action tokens in learned model. More flexible than Decision Transformer's return-conditioning: can compute value estimates, do MPC-style planning, etc. Foundation of transformer-based world models / offline RL.
Nash equilibrium in multi-agent RL — what and when? hard Policy profile ( π 1 , , π n ) ({\pi}_{1}, \;, \;{\pi}_{n}) ( π 1 , , π n ) In zero-sum games (Rock-Paper-Scissors, Chess, Go, StarCraft): minmax = maxmin = Nash. Solved via self-play (AlphaZero) or fictitious play + best-response. In mixed-motive / general-sum: multiple Nash equilibria + hard to find. Related concepts: correlated equilibrium, coarse correlated equilibrium (used in AlphaStar).
Fictitious play — what is it? hard Iterative: each iteration, each player computes best response to the empirical distribution of opponents' past strategies. Converges to Nash in zero-sum games (Robinson 1951). Foundation of self-play theory. Modern deep variants: NFSP (Neural Fictitious Self-Play, Heinrich & Silver 2016) — combines RL best response with supervised average-strategy learning. PSRO (Policy Space Response Oracle) generalizes further.
Options framework — formalization. hard Sutton, Precup, Singh 1999. An option = (I, π, β): initiation set I (states where the option can start), intra-option policy π, termination condition β(s) (probability of stopping). High-level policy chooses among options. Bellman equations extend naturally. Foundational abstraction for hierarchical RL. Modern: skill discovery methods (DIAYN, OPAL) auto-discover options from unsupervised interaction.
DPO derivation — the key mathematical insight. hard The optimal policy under KL-constrained reward max is π ⋅ ( y ∣ x ) {\pi} \cdot (y \mid x) π ⋅ ( y ∣ x ) π r e f ( y ∣ x ) {\pi}_{\mathrm{ref}}(y \mid x) π ref ( y ∣ x ) exp ( r ( x , y ) / β ) \operatorname{exp}(r(x, \;y) / {\beta}) exp ( r ( x , y ) / β ) Invert: r(x, y) = β log ( π ⋅ ( y ∣ x ) / π r e f ( y ∣ x ) ) + c o n s t \operatorname{log}({\pi} \cdot (y \mid x) / {\pi}_{\mathrm{ref}}(y \mid x))\; + \;\mathrm{const} log ( π ⋅ ( y ∣ x ) / π ref ( y ∣ x )) + const Substitute into Bradley-Terry preference model → loss depends only on π_θ, not r: L(θ) = -E [ log σ ( β log ( π θ ( y w ∣ x ) / π r e f ( y w ∣ x ) ) − β log ( π θ ( y l ∣ x ) / π r e f ( y l ∣ x ) ) ) ] E[\operatorname{log}\;{\sigma}({\beta}\;\operatorname{log}({\pi}{\theta}(y_{w} \mid x) / {\pi}_{\mathrm{ref}}(y_{w} \mid x))\; - \;{\beta}\;\operatorname{log}({\pi}{\theta}(y_{l} \mid x) / {\pi}_{\mathrm{ref}}(y_{l} \mid x)))] E [ log σ ( β log ( π θ ( y w ∣ x ) / π ref ( y w ∣ x )) − β log ( π θ ( y l ∣ x ) / π ref ( y l ∣ x )))] Skips reward model + PPO entirely; direct MLE on preference data.
Meta-RL — learning to learn. hard Train agent across a distribution of related tasks so it adapts to new tasks quickly. Approaches: (1) Recurrent meta-RL ( R L 2 ) (\mathrm{RL}^{2}) ( RL 2 ) (2) MAML for RL: gradient-based meta-learning that produces initialization enabling few-step adaptation. (3) PEARL: context-conditioned policy with latent task variable. Applications: fast robot adaptation, personalized recommenders. Modern LLM in-context learning is analogous.
How does transfer learning work in RL? medium Warm-start new policy from related-task policy (fine-tune) or pretrained representation. Techniques: (1) Progressive networks (Rusu et al.). (2) Distillation from teacher policy. (3) Pretrained visual encoders (VC-1, R3M) for vision-based robotics. (4) Task embedding + shared trunk. Modern: SL-pretrained transformer + RL fine-tune (LLM RLHF is transfer RL). Challenge: negative transfer if source task too different.
Successor features — what and why? hard Barreto et al. 2017. Decompose reward as r = φ(s, a, s') · w. Value function factors: V(s) = ψ(s) · w where ψ ( s ) = E [ Σ γ k φ ( s k , a k , s k ) ] {\psi}(s)\; = \;E[{\Sigma}\;{\gamma}^{k}\;{\varphi}(s_{k}, \;a_{k}, \;sk)] ψ ( s ) = E [ Σ γ k φ ( s k , a k , s k )] New task (new w) can reuse learned ψ — instant transfer without relearning value function. Foundation of generalized policy improvement (GPI) across tasks; applications in continual learning + universal RL.
Reward machines — structured reward specification. hard Represent complex non-Markovian rewards as finite-state automata: state tracks task progress (e.g., 'first find A, then find B'). Enables decomposition + credit assignment across sub-goals. Alternative to hand-crafted shaping. Foundation of LTL (Linear Temporal Logic) constrained RL. Applications: robotics multi-step tasks, curriculum learning. Modern extension: neuro-symbolic RL combining reward machines with NNs.
Potential-based reward shaping — why is it safe? hard Ng, Harada, Russell 1999. Add r'(s, a, s') = r(s, a, s') + γΦ(s') - Φ(s). Any potential function Φ is safe: doesn't change optimal policy (only value baseline). Because Σ t {\Sigma}_{t} Σ t γ t {\gamma}^{t} γ t ( γ Φ ( s t + 1 ) − Φ ( s t ) ) ({\gamma}{\Phi}(s_{t + 1})\; - \;{\Phi}(s_{t})) ( γ Φ ( s t + 1 ) − Φ ( s t )) t o − Φ ( s 0 ) + γ T \mathrm{to}\; - {\Phi}(s_{0})\; + \;{\gamma}^{T} to − Φ ( s 0 ) + γ T Φ ( s T ) {\Phi}(s_{T}) Φ ( s T ) Enables safe reward engineering without changing task. Standard technique in robotics reward design.
How do you handle partial observability in deep RL? medium (1) State-stacking: last k observations as input (DQN frame stack). (2) RNN / LSTM policy: hidden state summarizes history (R2D2). (3) Transformer over recent history (Decision Transformer). (4) Belief-state approximation via VAE or particle filter. (5) Reward-conditioned models. Rule: for weakly non-Markovian (short history matters), stacking + short RNN suffices. For long-horizon dependencies, transformer or explicit memory.
LQG — Linear Quadratic Gaussian control. hard Extension of LQR to noisy partial observations. Optimal controller = Kalman filter (state estimation) + LQR (control). Certainty equivalence: LQ control on filtered state estimate is optimal. Bridge between control theory + POMDP-RL. Modern: neural networks approximate both filter + controller. Foundation of classical control theory taught in every EE / ME curriculum.
Generalization in RL — why is it hard? hard Deep RL overfits to training environments — Cobbe et al. 2019 showed Atari-trained agents fail on procedurally-generated variants. Root causes: (1) narrow training distribution. (2) Memorization of trajectory-specific features. Fixes: (1) diverse env procedurally (ProcGen benchmark). (2) Regularization (dropout, weight decay). (3) Data augmentation. (4) Domain randomization. (5) Learn invariant / causal features. (6) Meta-learning. Ongoing challenge.
Catastrophic forgetting in continual RL. hard When switching between tasks or environments, RL agents forget previously learned skills. Root: SGD on new-task data overwrites old-task weights. Fixes: (1) EWC (Elastic Weight Consolidation): regularize toward important old-task weights. (2) Progressive networks: freeze old columns + add new. (3) Rehearsal: replay old experiences. (4) Modular networks: separate columns per task. (5) Meta-learning shared initialization. Ongoing research area.
Generalist agents — Gato and DeepMind's approach. hard Reed et al. 2022 (Gato): single transformer trained on 600+ tasks (Atari, robotics, chat, image captioning) as tokenized sequences. Uses cross-attention over task tokens. Modest per-task performance, but shows one architecture can handle many domains. Followed by RT-2 (robotics VLA), OpenVLA, PI-0 (Pi Zero) — multi-task foundation models for embodied AI. Modern trend: transformer + massive data across modalities.
Quantile regression in QR-DQN — the loss. hard Predict N quantiles τ i {\tau}_{i} τ i Loss on target y for quantile prediction Z i Z_{i} Z i τ i {\tau}_{i} τ i ( y − Z i ) (y\; - \;Z_{i}) ( y − Z i ) Total l o s s = Σ i \mathrm{loss}\; = \;{\Sigma}_{i} loss = Σ i E [ ρ τ i ( T D t a r g e t − Z i ) ] E[{\rho}{\tau}_{i}(\mathrm{TD}_{\mathrm{target}}\; - \;Z_{i})] E [ ρ τ i ( TD target − Z i )] Trains each quantile to correct level. Extension of C51 to continuous quantiles. Foundation of IQN.
Deadly triad — the three ingredients. hard (1) Function approximation (neural nets, not tabular). (2) Bootstrapping (TD target uses V, not full MC). (3) Off-policy data. Combining all three can cause divergence (Baird's counterexample: linear FA + off-policy TD → weights blow up). Modern deep RL usually has all three (DQN, SAC) — works empirically via replay + target nets + gradient clip. Understanding this triad is core RL theory.
Where is RL heading (2025+)? hard Trends: (1) RL for reasoning: verifier-based training becomes standard for math / code / science (o1, R1 continue). (2) Agent RL: WebArena / SWE-bench become primary post-training targets. (3) Multi-turn RL for LLMs: not just single response. (4) Test-time compute as first-class tuning knob. (5) Robotics VLA at scale (RT-2 successors). (6) World models finally matching model-free (Dreamer V3+). (7) Continual + open-ended learning. Frontier: superhuman capability via search + RL + LLM.
How do you choose the discount factor, and what goes wrong at the extremes? medium It sets the effective horizon, roughly one over one minus gamma steps, so pick it from how far ahead consequences actually matter in your problem rather than by convention. Too low and the agent is myopic: it ignores delayed reward entirely, which in a game looks like refusing to invest a move for a later gain. Too high and two things degrade, because the variance of returns grows with horizon length and the value function has to represent a much larger range, so bootstrapped estimates become unstable and slow to converge. In episodic tasks with a natural end, a value near one is defensible; in continuing tasks it is a variance knob you tune. A common practical trick is to train with a lower value and anneal it upward as the value function stabilizes.
What is the deadly triad and how do practical algorithms cope with it? hard The combination of function approximation, bootstrapping, and off-policy learning. Each is fine alone, but together they can make value estimates diverge, because a bootstrapped target is computed from the same approximator being updated, and off-policy data means the states are not distributed according to the policy whose values you are fitting. Deep Q-learning uses all three, which is why it needs the stabilizers it has: a target network to freeze the bootstrap target for a while, a replay buffer to decorrelate updates, and reward or gradient clipping to bound the magnitude. Double Q-learning additionally removes the maximization bias that makes divergence more likely. None of these are cosmetic; removing any one of them typically makes training visibly unstable.
Why can't you just run Q-learning on a fixed dataset? hard Because the maximization step queries actions the dataset never contains. The target takes a maximum over actions, and for unseen actions the network's value is an extrapolation with no data to correct it, so overestimation errors get selected precisely because they are overestimates. Bootstrapping then propagates those inflated values through the dataset, and the learned policy prefers exactly the actions nobody ever tried. Without environment interaction there is no corrective feedback loop, which is what makes offline learning qualitatively different from off-policy learning with a replay buffer. The fixes constrain the policy to the data: behaviour regularization, conservative value penalties on out-of-distribution actions, or filtered imitation of the best observed trajectories.