EasyDeepLearn
Reinforcement Learning · section 2 of 12

Value methods (Q-learning, DQN)

39 interview questions on value methods (q-learning, dqn), each answered in full. Free to read, no account needed.

What update does Q-learning perform?

medium
  • Q(s, a) <- Q(s, a) + alpha · (r  +  γ  ⋅  max⁡a  Q(s,  a)  −  Q(s,  a))(r\; + \;\gamma\; \cdot \;\operatorname{max}_{a}\;Q(s, \;a)\; - \;Q(s, \;a)).
  • This bootstraps toward the maximum future value at s', which is why it's off-policy: the target uses the greedy next action regardless of what the agent actually did.
  • With function approximation (DQN), you use a replay buffer and a target network to stabilize learning.
#value-methodsPermalink & quiz →

What are the key tricks that make DQN work?

hard
  • (1) Experience replay buffer: store transitions and sample randomly to break correlations and improve data efficiency.
  • (2) Target network: use a delayed copy of Q for the TD target, updated slowly, to stabilize training.
  • (3) Reward clipping and frame stacking (for Atari).
  • (4) Double DQN reduces overestimation bias.
  • (5) Prioritized replay samples surprising transitions more often.
  • (6) Dueling architecture separates value and advantage.
#value-methods#deep-rlPermalink & quiz →

Double DQN — what problem does it solve?

medium
  • Vanilla DQN target: max⁡a\operatorname{max}_{a} Qtarget(s,  a)Q_{\mathrm{target}}(s, \;a).
  • Same network selects AND evaluates max action → overestimation bias (noise favors overestimated actions).
  • Double DQN: use online network to select action, target network to evaluate.
  • Target  =  Qtarget(s,  argmax⁡a  Qonline(s,  a))\mathrm{Target}\; = \;Q_{\mathrm{target}}(s, \;\operatorname{argmax}_{a}\;Q_{\mathrm{online}}(s, \;a)).
  • Reduces overestimation, gives more stable learning.
  • Standard extension in modern DQN implementations.
#value-methods#deep-rlPermalink & quiz →

Dueling DQN — architecture.

hard
  • Split Q head into two streams: V(s) (state value) and A(s, a) (advantage), combined as Q(s,  a)  =  V(s)  +  A(s,  a)  −  meanaQ(s, \;a)\; = \;V(s)\; + \;A(s, \;a)\; - \;\mathrm{mean}_{a} A(s, a) (identifiability constraint).
  • Improves learning when many actions have similar values (V dominates); doesn't waste capacity redundantly encoding V per action.
  • Combined with Double DQN + prioritized replay + noisy nets → Rainbow DQN (Hessel et al. 2018).
#value-methods#deep-rlPermalink & quiz →

Prioritized replay — mechanism.

hard
  • Sample transitions with probability pip_{i} ∝ |δi{\delta}_{i}|^α (high TD error = more surprising = more informative).
  • Correct sampling bias with importance weights wi  =  (1/N  ⋅  1/Pi)w_{i}\; = \;(1 / N\; \cdot \;1 / P_{i})^β applied to the loss.
  • Improves data efficiency 2-3× on Atari.
  • Standard extension of DQN.
  • Modern replay libraries (segment tree implementation) allow O(log n) prioritized sampling.
#value-methods#deep-rlPermalink & quiz →

Rainbow DQN — what six extensions does it combine?

hard
  • (1) Double DQN, (2) Dueling architecture, (3) Prioritized replay, (4) Multi-step (n-step) returns, (5) Distributional RL (C51: model return distribution not just mean), (6) Noisy nets (parametric exploration replacing epsilon-greedy).
  • Hessel et al. 2018 showed each contributes; combined SOTA on Atari for its size.
  • Foundation of most subsequent value-based deep RL research.
#value-methods#deep-rlPermalink & quiz →

Distributional RL (C51, QR-DQN) — the idea.

hard
  • Model the FULL distribution of returns Z(s, a) instead of just its mean Q(s, a) = E[Z(s, a)].
  • C51 (Bellemare et al. 2017): discretize into 51 atoms, learn categorical distribution over returns.
  • QR-DQN: quantile regression over N quantiles.
  • IQN: implicit quantile network.
  • Benefits: richer signal, better risk-sensitivity, empirically better performance.
  • Distributional Bellman: Z(s, a) = R(s, a) + γ Z(s,  argmax⁡a  Q)Z(s, \;\operatorname{argmax}_{a}\;Q).
#value-methods#deep-rlPermalink & quiz →

Noisy nets — how do they replace ε-greedy?

hard
  • Add parametric noise to weights of last layer(s): W = μ + σ ⊙ ε with learnable μ, σ.
  • Exploration comes from state-dependent noise, learned per-parameter.
  • As σ shrinks, exploration decreases automatically → no manual ε schedule.
  • Used in Rainbow DQN.
  • Advantage: more directed exploration than uniform ε-greedy; noise magnitude adapts to what the network needs.
#exploration#deep-rlPermalink & quiz →

SARSA update — on-policy analog of Q-learning.

medium
  • Q(s, a) ← Q(s, a) + α [r + γ Q(s', a') - Q(s, a)] where a' is the ACTUAL action taken from s' (sampled from current π).
  • On-policy: target uses the policy's next action, not the greedy one.
  • More conservative than Q-learning: converges to Q for the policy including its exploration (safer near cliffs).
  • Expected SARSA: use E_π[Q(s', a')] instead of sampled a' → less variance.
#value-methodsPermalink & quiz →

How do you size a replay buffer?

medium
  • Trade-off: bigger buffer → more diverse data, older stale off-policy data.
  • Rule: enough to cover diverse experience but not so large that stale data hurts.
  • Atari DQN: 1M transitions.
  • Continuous control: 106−10710^{6} - 10^{7}.
  • Off-policy sensitivity: SAC works with larger buffers; DDPG less so.
  • Prioritized replay adjusts distribution automatically.
  • Off-policy correction (V-trace, ReTrace) helps with staleness in on-policy-ish algorithms.
#value-methods#engineeringPermalink & quiz →

Hard update vs soft (Polyak) update for target networks.

medium
  • Hard: copy θtarget{\theta}_{\mathrm{target}} ← θ every N steps (DQN: every 10k).
  • Simple.
  • Soft: θtarget{\theta}_{\mathrm{target}} ← τθ + (1-τ) θtarget{\theta}_{\mathrm{target}} every step (SAC/DDPG: τ = 0.005).
  • Smoother, avoids sudden target shifts.
  • Both stabilize TD training.
  • Rule: hard update for large jumps that matter (discrete DQN), soft for continuous / actor-critic settings.
  • Modern default is soft update with τ ~ 0.001-0.01.
#value-methods#engineeringPermalink & quiz →

Why does DQN scale poorly to continuous action spaces?

hard
  • Q-learning requires argmax⁡a\operatorname{argmax}_{a} Q(s, a) — over a continuous action space, this is a nested optimization at every step (expensive) or requires discretizing (loses fidelity).
  • Solutions: (1) Actor-critic (DDPG, SAC): explicit π_θ(s) network outputs continuous action; (2) NAF (Normalized Advantage Function): analytic argmax via quadratic-in-a parameterization; (3) sample-based action selection with CEM (QT-Opt).
  • Modern default: SAC + reparameterized Gaussian policy.
#value-methods#deep-rlPermalink & quiz →

Fitted Q-Iteration — batch off-policy Q-learning.

hard
  • Alternate: (1) collect / accumulate dataset D of (s, a, r, s') transitions, (2) fit Qk+1Q^{k + 1} to targets yi  =  ri  +  γy_{i}\; = \;r_{i}\; + \;{\gamma} max⁡a\operatorname{max}_{a} Qk(si,  a)Q^{k}(si, \;a) via supervised regression (any regressor: tree, NN).
  • Repeat.
  • Batch-mode, off-policy, works with existing logged data.
  • Foundation of many offline RL methods and NFQ (Neural Fitted Q).
  • DQN = FQI + online replay + target net.
#value-methods#offline-rlPermalink & quiz →

C51 — how does the projected Bellman update work?

hard
  • Distribution over N=51 atoms ziz_{i} in [Vmin,  Vmax][V_{\mathrm{min}}, \;V_{\mathrm{max}}] with probability pi(s,  a)p_{i}(s, \;a).
  • Bellman: sample transition, compute target atom ziz_{i}' = clip(r  +  γ  zi,  Vmin,  Vmax)\mathrm{clip}(r\; + \;{\gamma}\;z_{i}, \;V_{\mathrm{min}}, \;V_{\mathrm{max}}); this may not land on a valid atom → distribute probability mass proportionally to two nearest atoms (linear interpolation).
  • Loss: KL divergence between projected target distribution and current Q distribution.
  • Elegant, richer signal than Q mean-value.
#value-methods#deep-rlPermalink & quiz →

IQN — Implicit Quantile Networks.

hard
  • Instead of predicting discrete atoms (C51) or fixed quantiles (QR-DQN), IQN learns to output ANY quantile τ ∈ [0, 1] given as input to the network.
  • Input concatenates state + embedding of τ.
  • Enables sampling arbitrary many quantiles at inference for risk-sensitive control.
  • Cleaner formulation than QR-DQN's fixed quantiles.
  • State-of-the-art distributional RL circa 2018-2020.
#value-methods#deep-rlPermalink & quiz →

Why is naive DQN unstable and what specific fixes address which cause?

hard
  • (1) Correlated sequential data → replay buffer breaks correlation.
  • (2) Moving target Q'-target = Q → target network freezes target for N steps.
  • (3) Overestimation from noise in max → Double DQN.
  • (4) Reward magnitude / scale → reward clipping to [-1, 1] or reward normalization.
  • (5) Sparse reward → n-step returns or intrinsic curiosity.
  • Every DQN implementation combines at least (1) + (2); Rainbow adds all six.
#value-methods#deep-rl#engineeringPermalink & quiz →

Value decomposition (VDN, QMIX) — for multi-agent Q-learning.

hard
  • In cooperative multi-agent, joint Q(s,  a1,  ,  an)Q(s, \;a_{1}, \;, \;a_{n}) grows exponentially.
  • VDN: Qtotal  =  ΣQ_{\mathrm{total}}\; = \;{\Sigma} Qi(s,  ai)Q_{i}(s, \;a_{i}) → decentralized execution + centralized training.
  • QMIX: Qtotal  =  fmix(Q1,  ,  Qn  s)Q_{\mathrm{total}}\; = \;f_{\mathrm{mix}}(Q_{1}, \;, \;Q_{n}\;s) with monotonic mixing network (∂f/∂Qi  ≥  0)( \partial f / \partial Q_{i}\; \ge \;0) → richer than VDN, still allows decentralized argmax.
  • Foundation of centralized-training-decentralized-execution paradigm for cooperative multi-agent RL.
#value-methods#multi-agentPermalink & quiz →

DDPG — Deep Deterministic Policy Gradient.

hard
  • Off-policy actor-critic for continuous actions.
  • Deterministic policy μ_θ(s) + Q-critic Q_φ(s, a).
  • Actor gradient: ∇_θ E[Q(s, μ(s))] via deterministic policy gradient theorem.
  • Uses replay buffer + soft target networks + OU noise for exploration.
  • Sensitive to hyperparameters + Q-overestimation issues → superseded by TD3 and SAC.
  • Historic importance as first successful continuous-action deep RL.
#policy-methods#actor-critic#deep-rlPermalink & quiz →

TD3 — Twin Delayed DDPG improvements over DDPG.

hard
  • Fujimoto et al. 2018 fixes: (1) Twin critics: two Q-networks, use min for target → mitigates overestimation.
  • (2) Delayed policy updates: update actor every 2 critic steps → gives critic time to catch up.
  • (3) Target policy smoothing: add clipped noise to target actions → reduces exploitation of Q's peaks.
  • Result: much more stable than DDPG on continuous control.
  • Standard baseline before SAC.
#policy-methods#actor-critic#deep-rlPermalink & quiz →

SAC — Soft Actor-Critic mechanism and advantages.

hard
  • Maximum entropy RL: maximize E[Σ  γt  (rt  +  α  H(π(⋅∣st)))]E[{\Sigma}\;{\gamma}^{t}\;(r_{t}\; + \;{\alpha}\;H({\pi}( \cdot \mid s_{t})))] where H is policy entropy.
  • Adds entropy bonus → exploration + robustness.
  • Off-policy, stochastic policy (reparameterized Gaussian), twin Q-critics (TD3-style).
  • Automatically tunes α to hit target entropy.
  • Robust across many continuous-control tasks with default hyperparams.
  • Modern default for continuous action deep RL (Haarnoja 2018).
#policy-methods#actor-critic#deep-rlPermalink & quiz →

IMPALA — how does it scale actor-critic?

hard
  • Actor-learner architecture: many parallel actor threads roll out trajectories using slightly-stale policy, single learner updates.
  • Off-policy correction via V-trace (Espeholt et al. 2018): truncated importance sampling with clipping.
  • Enables massive scaling (thousands of actors).
  • Foundation of distributed deep RL (used in AlphaStar, OpenAI Five).
  • Trades on-policyness for throughput + parallel scaling.
#actor-critic#deep-rl#engineeringPermalink & quiz →

Vectorized environments — why and how?

medium
  • Batch multiple env instances that step in parallel per learner step → wider (batched) but same-length trajectory data.
  • Benefits: (1) better GPU utilization for policy inference (batch forward pass over N envs), (2) more diverse data per update, (3) faster wall-clock time.
  • Standard: gym.vector.SyncVectorEnv (CPU-bound) or AsyncVectorEnv (parallel workers), OR IsaacGym / Envpool for massively-parallel simulation on GPU.
#engineering#deep-rlPermalink & quiz →

Bootstrapped DQN — approximate Thompson sampling for deep RL.

hard
  • K parallel Q-heads sharing a body, each trained on a different bootstrap of the replay buffer (mask indicates which head learns from which transition).
  • At episode start, sample one head, act greedy w.r.t. it for entire episode → deep exploration (temporally-extended commitment).
  • Beats ε-greedy on long-horizon exploration tasks (Deep Sea, hard-exploration Atari).
#exploration#deep-rlPermalink & quiz →

Count-based exploration for large state spaces.

hard
  • Add intrinsic reward rint  =  βr_{\mathrm{int}}\; = \;{\beta} / √N(s) where N(s) = visit count.
  • Naive count fails in continuous / high-dim state → use pseudo-counts from density model (Bellemare et al. 2016) or hashing (Tang et al. 2017).
  • Encourages visiting rare states.
  • Foundation of hard-exploration deep RL.
  • Modern successor: RND (Random Network Distillation) — cheaper + effective.
#exploration#deep-rlPermalink & quiz →

Random Network Distillation (RND) — mechanism.

hard
  • Fixed random target network ftarget(s)f_{\mathrm{target}}(s) initialized randomly + trainable predictor fpred(s)f_{\mathrm{pred}}(s).
  • Intrinsic reward = ||fpred(s)  −  ftarget(s)f_{\mathrm{pred}}(s)\; - \;f_{\mathrm{target}}(s)||².
  • Predictor learns quickly on frequently-visited states → low intrinsic reward.
  • Rare states have high error → high intrinsic reward.
  • Simple, cheap, effective — solved Montezuma's Revenge (Burda et al. 2018).
  • Foundation of modern exploration methods.
#exploration#deep-rlPermalink & quiz →

Intrinsic Curiosity Module (ICM) — Pathak et al.

hard
  • Learn forward model f(s, a) → predicted φ(st+1){\varphi}(s_{t + 1}) and inverse model g(st,  st+1)g(s_{t}, \;s_{t + 1}) → ata_{t}.
  • Encode state via φ trained by inverse loss (predicts a from state pair) → learns features that matter for control.
  • Intrinsic reward = ||f(s, a) - φ(s')||² = model prediction error → curiosity toward unpredictable transitions.
  • Inverse-model feature learning is key for filtering out irrelevant randomness ('noisy TV' problem).
#exploration#deep-rlPermalink & quiz →

Go-Explore — how does it solve hard-exploration?

hard
  • Ecoffet et al. 2019, 2021: (1) archive all visited states (cell-hashed).
  • (2) 'Go' phase: teleport back to promising archived state (or replay actions).
  • (3) 'Explore' phase: random exploration from that state; if new state discovered, add to archive.
  • (4) Later: distill trajectories into robust policy via imitation learning.
  • Broke Montezuma's Revenge + Pitfall.
  • Insight: separate remembering promising states from acting from them.
#exploration#deep-rlPermalink & quiz →

Hindsight Experience Replay (HER) — the trick.

hard
  • Andrychowicz et al. 2017: for a failed episode aiming at goal g, re-label the trajectory as if the final state achieved was the goal.
  • Failed attempts become successful demonstrations of reaching that alternative goal.
  • Enables learning from sparse-reward goal-conditioned tasks.
  • Standard in robotics goal-reaching (pick and place, manipulation).
  • Combined with DDPG / SAC for continuous control.
#exploration#deep-rlPermalink & quiz →

Agent57 — how did it achieve above-human on all Atari?

hard
  • Combined: (1) meta-controller balancing exploration ↔ exploitation across an episode by choosing among a family of policies with different exploration weights.
  • (2) NGU (Never Give Up) intrinsic motivation: episodic + long-term.
  • (3) Distributed R2D2 backbone (recurrent DQN + distributed).
  • Reached human-level on all 57 Atari games (Puigdomènech et al. 2020).
  • Milestone in general RL.
#exploration#deep-rlPermalink & quiz →

World models (Ha & Schmidhuber, Dreamer) — big idea.

hard
  • Learn a compact latent dynamics model of the environment: encoder + RNN dynamics + reward predictor.
  • Train policy inside the model ('imagination') rather than the real env.
  • Massive sample efficiency: real env only needed to keep the model calibrated.
  • Dreamer V3: SOTA on many benchmarks with modest data.
  • Foundation of modern model-based deep RL.
  • Also enables offline planning by rolling out counterfactuals.
#model-based#planning#deep-rlPermalink & quiz →

MuZero — model-based RL without knowing the env dynamics.

hard
  • Schrittwieser et al. 2020: learn a representation + dynamics + reward + value network end-to-end from experience → MCTS in latent space.
  • No need for environment simulator (unlike AlphaZero).
  • Same NN outputs policy prior + value estimates.
  • Trained via combination of TD, MC-return matching, and MCTS policy matching.
  • Superhuman Chess/Go/Shogi + top on 57 Atari with a single method.
#model-based#planning#deep-rlPermalink & quiz →

MBPO — Model-Based Policy Optimization.

hard
  • Janner et al. 2019: use an ensemble of forward models.
  • Every real step, generate k short imagined rollouts (branch factor: many rollouts, short length ~ 1-15 steps).
  • Train off-policy actor-critic (SAC) on mixture of real + imagined.
  • Ensembles give uncertainty → keep rollout short where model unreliable.
  • Reaches model-free performance with 10-100× fewer real interactions.
  • Key MBRL milestone.
#model-based#deep-rlPermalink & quiz →

How do you improve sample efficiency in deep RL?

medium
  • (1) Replay buffer + off-policy learning (DQN, SAC).
  • (2) Model-based (Dreamer, MBPO).
  • (3) Higher update-to-data ratio (many gradient steps per env step).
  • (4) Auxiliary losses on encoder (representation learning during RL — SPR, CURL).
  • (5) Transfer from pre-trained representations (VC-1, R3M for vision robotics).
  • (6) Regularize to expert demos (BC + RL).
  • Robotics: (7) sim-to-real to leverage cheap simulation.
#deep-rl#engineeringPermalink & quiz →

Sim-to-real — main techniques.

hard
  • (1) Domain randomization: randomize physics params (mass, friction, latency) in sim → policy robust to real variation.
  • (2) Domain adaptation: fine-tune on real data.
  • (3) System identification: fit sim params to match real data.
  • (4) Adversarial DR / meta-learning: adaptive randomization.
  • (5) Photorealistic sim + vision randomization.
  • Standard in robotics locomotion (Tan et al. 2018, Rudin et al. 2022) + manipulation (OpenAI's Rubik's cube).
#deep-rl#applicationsPermalink & quiz →

R2D2 — distributed recurrent DQN.

hard
  • Kapturowski et al. 2019: recurrent Q-network (LSTM) for partial observability + prioritized replay with sequence-based sampling + distributed actor-learner + burn-in prefix (feed part of sequence with old hidden state before computing loss).
  • Combines R (recurrence) + D (distributed) + D (double / dueling).
  • Foundational for hard-memory Atari + Agent57.
#value-methods#deep-rlPermalink & quiz →

Why does DQN stack 4 consecutive frames?

easy
  • Atari observation is a single frame — insufficient for Markov state (need velocity: ball direction, projectile motion).
  • Stack 4 last frames as input → agent can infer motion.
  • Alternative: recurrent Q-net (R2D2) learns temporal features directly.
  • Also: reward and terminal from previous step included.
  • Standard preprocessing: grayscale, downscale to 84×84, frame-skip 4 (act every 4 frames).
  • Foundation of DQN Atari benchmark.
#deep-rl#engineeringPermalink & quiz →

Reward clipping in DQN — trade-offs.

medium
  • DQN Atari clips rewards to {-1, 0, +1} → stabilizes Q-learning across games with vastly different reward scales (Ms. Pac-Man vs Skiing).
  • Cost: agent doesn't distinguish 1-point vs 100-point reward → suboptimal for scale-sensitive tasks.
  • Modern replacement: reward normalization by running SD (used in PPO), or distributional RL that models absolute return distribution.
  • Best of both worlds: keep scale info but normalize for gradients.
#deep-rl#engineeringPermalink & quiz →

OpenAI Five — Dota 2 milestone.

hard
  • Five-agent RL (one per hero) trained via self-play on massive scale (128k CPU cores, 256 GPUs).
  • PPO with LSTM policy, shared parameters across heroes with role embedding.
  • Reward shaping via team + individual signals.
  • Coordination is emergent — no explicit coordination mechanism.
  • Defeated world-champion team OG in 2019.
  • Foundation of large-scale team-game MARL.
#multi-agent#deep-rlPermalink & quiz →

Plasticity loss in deep RL — the problem.

hard
  • Deep RL agents lose the ability to learn new patterns after long training — 'plasticity loss'.
  • Nikishin et al. 2022: root cause is weight rank collapse + dead ReLU units accumulating.
  • Fixes: (1) periodic reset of top layers (primacy bias reduction).
  • (2) Regenerative regularization (RegLoss).
  • (3) Layer norm + orthogonal reinitialization.
  • (4) Continual learning tricks.
  • Critical for long-horizon RL training + continual scenarios.
#deep-rl#engineeringPermalink & quiz →

Practise Reinforcement Learning

All 12 sectionsFlashcardsQuiz