EasyDeepLearn
Reinforcement Learning · section 3 of 12

Policy gradients & actor-critic

25 interview questions on policy gradients & actor-critic, each answered in full. Free to read, no account needed.

What is a policy gradient and why do we use it?

hard
  • A policy gradient directly optimizes the parameters theta of a stochastic policy πθ(a∣s)\pi_{\theta}(a \mid s) to maximize expected return: grad J  =  E[grad  log⁡  πθ(a∣s)  ⋅  A(s,a)]J\; = \;E[\mathrm{grad}\;\operatorname{log}\;\pi_{\theta}(a \mid s)\; \cdot \;A(s, a)].
  • Advantages: handles continuous action spaces, learns stochastic policies (crucial for partial observability), and integrates naturally with neural networks.
  • Disadvantages: high variance — you need baselines/advantage estimation to stabilize.
#policy-methodsPermalink & quiz →

What is an actor-critic algorithm?

medium
  • Actor-critic combines a policy (actor) that picks actions and a value function (critic) that estimates the advantage or value.
  • The critic reduces the variance of the policy-gradient estimate.
  • Popular variants: A2C/A3C, PPO (clipped surrogate objective, on-policy), and SAC (off-policy, entropy-regularized).
  • PPO is the workhorse for RLHF of language models.
#policy-methods#actor-criticPermalink & quiz →

Why is PPO so widely used?

medium
  • PPO is a policy-gradient method with a clipped surrogate objective that prevents the policy from moving too far from the old one at each update.
  • It is on-policy, simple to implement, robust across environments with default hyperparameters, and empirically competitive with more complex methods.
  • It is the standard algorithm for RLHF of large language models.
#policy-methods#actor-criticPermalink & quiz →

REINFORCE — the vanilla policy-gradient algorithm.

medium
  • For episode τ with return G(τ), update θ ← θ + α G(τ) ∇_θ log π_θ(at  ∣  st)(a_{t}\; \mid \;s_{t}) summed over trajectory.
  • Unbiased gradient of expected return via log-derivative trick.
  • Very high variance: rewards accumulate over full episode → large gradient variance.
  • Fixes: (1) subtract baseline b(s) (state-dep baseline / value function), (2) use advantage A(s, a) instead of raw G, (3) reward-to-go instead of full return.
#policy-methodsPermalink & quiz →

TRPO — Trust Region Policy Optimization.

hard
  • Constrained optimization: max_θ E[πθ/πold  ⋅  A]E[{\pi}{\theta} / {\pi}_{\mathrm{old}}\; \cdot \;A] s.t.
  • E[KL⁡(πold  ∣∣  πθ)]E[\operatorname{KL}({\pi}_{\mathrm{old}}\; \mid \mid \;{\pi}{\theta})] ≤ δ.
  • Solved via conjugate gradient + line search on the natural gradient direction.
  • Guarantees monotonic policy improvement (theoretically).
  • Complex to implement + expensive per update.
  • PPO simplifies this to a clipped objective (much easier, similar performance) → PPO is basically TRPO's practical successor.
#policy-methods#actor-criticPermalink & quiz →

PPO's clipped surrogate objective — write it and explain.

hard
  • LCLIP(θ)  =  E[min⁡(rt(θ)  At,  clip(rt(θ),  1−ε,  1+ε)  At)]L^{\mathrm{CLIP}}({\theta})\; = \;E[\operatorname{min}(r_{t}({\theta})\;A_{t}, \;\mathrm{clip}(r_{t}({\theta}), \;1 - {\varepsilon}, \;1 + {\varepsilon})\;A_{t})] where rt(θ)  =  πr_{t}({\theta})\; = \;{\pi}_θ(a∣s)  /  πold(a∣s)(a \mid s)\; / \;{\pi}_{\mathrm{old}}(a \mid s).
  • Clip stops the policy from moving too far from πold{\pi}_{\mathrm{old}} on either side. min ensures we take the more conservative estimate.
  • Typical ε = 0.2.
  • Combined with GAE advantage and clipped value loss → PPO2 (standard).
  • Simpler and works better in practice than TRPO's constrained optimization.
#policy-methods#actor-criticPermalink & quiz →

PPO implementation details that actually matter.

hard
  • Engstrom et al. 2020 'Implementation Matters': (1) value function clipping, (2) reward scaling by running SD, (3) orthogonal init, (4) Adam with fixed lr (β2  =  0.999)({\beta}_{2}\; = \;0.999), (5) global gradient clip to 0.5, (6) GAE with λ = 0.95, (7) normalize advantages per-batch, (8) LR annealing, (9) N epochs per rollout (~4-10).
  • Many published gains are due to these tricks, not the core algorithm.
  • Always study a modern reference implementation.
#policy-methods#engineeringPermalink & quiz →

A2C vs A3C — the difference.

medium
  • A3C (Asynchronous Advantage Actor-Critic, Mnih 2016): multiple async workers each with own env, updating shared params — lock-free async gradient updates.
  • A2C (synchronous): workers step in parallel, batch gradients synchronously, single update per batch — better GPU utilization, more reproducible.
  • A2C is simpler and empirically as good; A3C was pre-GPU thinking.
  • Modern default: A2C or PPO on batched vectorized envs.
#policy-methods#actor-criticPermalink & quiz →

Why add an entropy bonus to the policy objective?

medium
  • Encourages stochastic exploration + prevents premature convergence to suboptimal deterministic policy.
  • Loss = -E[log π · A] - β H(π).
  • Also gives robustness (multiple near-optimal actions).
  • Foundation of SAC (with automatic α tuning) and A3C-style entropy regularization.
  • Trade-off: too high β → agent won't commit; too low → early exploitation.
  • In LLM RLHF: entropy prevents mode collapse toward one response.
#policy-methods#explorationPermalink & quiz →

PPO vs SAC — which do you pick?

medium
  • PPO: on-policy, simple to implement + tune, works everywhere reasonably, standard for RLHF of LLMs. SAC: off-policy, sample-efficient (uses replay), maximum entropy → robust, best for continuous control (robotics).
  • Rule: (1) discrete + easy tuning + big rollouts → PPO.
  • (2) Continuous + sample efficiency matters → SAC.
  • (3) Large-scale distributed → PPO with IMPALA-style actors.
  • (4) LLM alignment → PPO (RLHF standard).
#policy-methods#actor-criticPermalink & quiz →

DPO — Direct Preference Optimization instead of RLHF PPO.

hard
  • Rafailov et al. 2023: skips explicit reward model + PPO.
  • Direct loss on preference pairs: L(θ) = -E[log⁡  σ(β  log⁡  πθ(yw∣x)/πref(yw∣x)  −  β  log⁡  πθ(yl∣x)/πref(yl∣x))]E[\operatorname{log}\;{\sigma}({\beta}\;\operatorname{log}\;{\pi}{\theta}(y_{w} \mid x) / {\pi}_{\mathrm{ref}}(y_{w} \mid x)\; - \;{\beta}\;\operatorname{log}\;{\pi}{\theta}(y_{l} \mid x) / {\pi}_{\mathrm{ref}}(y_{l} \mid x))].
  • Derived from KL-constrained reward-maximization → closed-form optimal policy in terms of reward → invert to remove reward.
  • Simpler + more stable than PPO.
  • Now standard for open-source LLM alignment (Llama, Zephyr).
#llm#alignment#policy-methodsPermalink & quiz →

GRPO — Group Relative Policy Optimization (DeepSeekMath, R1).

hard
  • PPO variant that skips the value network.
  • For each prompt, sample G responses, compute rewards, normalize as advantages: Ai  =  (ri  −  meanr)  /  stdrA_{i}\; = \;(r_{i}\; - \;\mathrm{mean}_{r})\; / \;\mathrm{std}_{r}.
  • Uses group-normalized rewards as advantage estimates → no critic needed.
  • Preserves PPO's clipped surrogate loss.
  • Enables efficient RL fine-tuning of reasoning models (DeepSeek-R1).
  • Modern efficiency win for LLM RL.
#llm#alignment#policy-methodsPermalink & quiz →

Categorical vs Gaussian vs beta policies — which and when?

medium
  • Categorical: discrete actions, softmax over logits.
  • Gaussian: continuous, N(μθ(s),  σθ(s)2)N({\mu}{\theta}(s), \;{\sigma}{\theta}(s)^{2}). σ can be state-dependent or state-independent parameter.
  • Beta: bounded continuous [a, b] → avoids clipping bias of unbounded Gaussian.
  • Tanh-Gaussian (squashed): Gaussian pass through tanh → bounded.
  • Modern default: tanh-Gaussian (SAC uses it).
  • Beta useful for asymmetric action bounds.
#policy-methodsPermalink & quiz →

Value function clipping in PPO — why and how?

hard
  • LVF  =  max⁡L^{\mathrm{VF}}\; = \;\operatorname{max}((Vθ(st)  −  Vtarg)2(V{\theta}(s_{t})\; - \;V_{\mathrm{targ}})^{2}, (clip(Vθ(st),  Vold  −  ε,  Vold  +  ε)  −  Vtarg\mathrm{clip}(V{\theta}(s_{t}), \;V_{\mathrm{old}}\; - \;{\varepsilon}, \;V_{\mathrm{old}}\; + \;{\varepsilon})\; - \;V_{\mathrm{targ}})²) → same trust-region-style clip for the value function.
  • Prevents value update from swinging wildly per epoch.
  • Modest empirical improvement + more stable.
  • Implementation detail buried in original PPO paper's appendix but has real impact.
#actor-critic#engineeringPermalink & quiz →

How does batch size / rollout length affect PPO?

medium
  • Larger rollout length → lower variance advantage estimates (more MC-like), but slower feedback.
  • Batch size (nactors  ×  rolloutlength)(n_{\mathrm{actors}}\; \times \;\mathrm{rollout}_{\mathrm{length}}): must be big enough for a stable gradient estimate; typical 2048-16384.
  • Too small → noisy gradients + unstable KL.
  • Too big → wasted computation.
  • Modern LLM RLHF uses very large batch sizes (millions of tokens per update).
#policy-methods#engineeringPermalink & quiz →

Ornstein-Uhlenbeck noise vs Gaussian noise for continuous control.

medium
  • OU: temporally correlated noise: dxt  =  θ(μ  −  xt)\mathrm{dx}_{t}\; = \;{\theta}({\mu}\; - \;x_{t})dt + σ dWt\mathrm{dW}_{t} → mean-reverting Brownian.
  • Used in original DDPG for exploration on physical control tasks — smooth trajectories match physical inertia.
  • Later work (TD3, SAC) uses simple Gaussian action noise with equal performance.
  • Modern default: Gaussian noise; OU is historical / niche.
#exploration#actor-criticPermalink & quiz →

MADDPG — Multi-Agent DDPG.

hard
  • Lowe et al. 2017.
  • Each agent has its own actor πi(oi){\pi}_{i}(o_{i}) and critic Qi(s,  a1,  ,  an)Q_{i}(s, \;a_{1}, \;, \;a_{n}) (centralized: sees ALL agents' actions).
  • Actor uses only local obs, critic uses joint info → CTDE.
  • Handles cooperative + competitive + mixed.
  • Foundational cooperative-continuous-action MARL algorithm.
  • Modern variants: MATD3, MASAC.
#multi-agent#actor-criticPermalink & quiz →

MAPPO — Multi-Agent PPO.

medium
  • Yu et al. 2022.
  • Simply PPO with a centralized value function using joint observations, shared across cooperative agents.
  • Surprisingly strong: matches or beats specialized MARL algorithms (QMIX, MADDPG) on many benchmarks (SMAC, MPE) with careful implementation.
  • Foundation of 'PPO is enough' school of MARL.
  • Simpler than QMIX/MADDPG; recommended default.
#multi-agent#actor-criticPermalink & quiz →

Hierarchical RL — why and how?

hard
  • Decompose long-horizon tasks into (1) high-level policy over sub-goals or options, (2) low-level policy executing skills.
  • Advantages: temporal abstraction (reason over many steps at high level), sample efficiency, transfer of skills.
  • Frameworks: Options (Sutton, Precup, Singh), FeUdal Networks (Vezhnevets), HIRO (goal-conditioned low-level), option-critic architecture.
  • Modern: hierarchical planners in LLM agents.
#policy-methods#planningPermalink & quiz →

DIAYN — Diversity is All You Need for skill discovery.

hard
  • Eysenbach et al. 2018.
  • Unsupervised skill learning: (1) sample skill z from prior, (2) train π(a  ∣  s,  z){\pi}(a\; \mid \;s, \;z) to maximize mutual information between z and visited states s: max I(s; z).
  • Uses a discriminator D(z  ∣  s)D(z\; \mid \;s) trained to identify skill from state.
  • Yields diverse skills without external reward.
  • Foundation of unsupervised RL and skill libraries for downstream tasks.
#exploration#policy-methodsPermalink & quiz →

Goal-conditioned RL — setup.

medium
  • Policy π(a  ∣  s,  g){\pi}(a\; \mid \;s, \;g) conditioned on desired goal g.
  • Reward r(s, a, g) = f(distance to g) or -1 until g reached.
  • Enables one policy for many tasks.
  • Combined with HER (relabel failed attempts) for sample efficiency.
  • Foundation of universal value function approximation (UVFA, Schaul 2015).
  • Modern uses: robot manipulation reaching diverse target poses, hierarchical HIRO planning, generalist RL agents.
#policy-methodsPermalink & quiz →

Interview: 'explain why PPO is on-policy but uses a ratio to importance-sample.'

hard
  • On-policy in spirit: PPO uses only recent data collected under πold{\pi}_{\mathrm{old}} (close  to  πθ)(\mathrm{close}\;\mathrm{to}\;{\pi}{\theta}).
  • The importance ratio r = π_θ / πold{\pi}_{\mathrm{old}} accounts for the fact that after several gradient steps within the epoch, π_θ has drifted from the data-generating πold{\pi}_{\mathrm{old}} — but not far (clipping enforces this).
  • Compare: fully off-policy (SAC, Q-learning) reuses ancient data.
  • PPO's 'small IS correction' is the whole point of the clipped surrogate.
#interview#policy-methodsPermalink & quiz →

Curriculum learning in RL — how to design one?

medium
  • Order training from easy to hard tasks: (1) start with dense reward, gradually sparsify.
  • (2) Start with easy env, add complexity (obstacles, distractors).
  • (3) Adaptive: automatic difficulty tuning to keep learning at frontier of ability (POET, ADR).
  • (4) Self-play induces natural curriculum via opponent's growing skill.
  • (5) Sub-goal decomposition: reach intermediate states before final.
  • Modern: adversarial curriculum where env teacher proposes tasks the student can barely solve.
#policy-methods#engineeringPermalink & quiz →

What is the clipping in PPO actually protecting you from?

hard
  • From a policy update so large that the data you collected no longer describes the policy you now have.
  • Policy gradient estimates are only valid near the behaviour policy, so a big step invalidates the very samples that justified it, and the classic symptom is a policy that collapses after a promising run.
  • Clipping the probability ratio removes the incentive to move a given action's probability beyond a trust region, so the objective flattens instead of rewarding ever larger changes.
  • It is a cheap surrogate for the constrained optimization TRPO solves exactly, trading theoretical guarantees for a first-order method that fits in a few lines.
  • It does not prevent divergence on its own, which is why implementations also limit epochs per batch and monitor the KL divergence.
#policy-methods#actor-criticPermalink & quiz →

When does the sample inefficiency of on-policy methods stop mattering?

medium
  • When samples are cheap and stability is expensive, which is the case with a fast parallelizable simulator.
  • If you can run thousands of environment steps per second across many workers, PPO's need for fresh data on every update costs wall-clock time you have, and you get back a method that is markedly less brittle and has far fewer hyperparameters that can silently destroy a run.
  • Off-policy methods such as SAC reuse a replay buffer and can be an order of magnitude more sample efficient, which is what you need when each interaction involves a real robot, a real user, or a slow physics engine.
  • The decision therefore rests on the cost of an environment step relative to the cost of engineering time spent stabilizing training.
#policy-methods#engineeringPermalink & quiz →

Practise Reinforcement Learning

All 12 sectionsFlashcardsQuiz