EasyDeepLearn
Reinforcement Learning · section 7 of 12

Deep RL engineering

7 interview questions on deep rl engineering, each answered in full. Free to read, no account needed.

Learning-rate schedule for PPO / SAC — what works?

medium
  • PPO: linear anneal from 3e-4 to 0 over training.
  • Adam optimizer, β2  =  0.999{\beta}_{2}\; = \;0.999.
  • SAC: fixed 3e-4 typically.
  • LLM RLHF: much smaller LR (1e-6 to 1e-5) since starting from a well-trained model; sometimes with cosine schedule.
  • Never use aggressive schedules that spike — RL is fragile to LR.
  • Warmup helps prevent initial destabilization when starting fresh.
#engineeringPermalink & quiz →

Population-Based Training (PBT) for RL.

hard
  • Jaderberg et al. 2017: run N agents in parallel with different hyperparameters.
  • Periodically: worst agents copy weights from top-N + perturb hyperparameters.
  • Simultaneously trains agents AND does hyperparameter optimization — no manual sweep.
  • Standard in complex RL (DeepMind's Capture-the-Flag, AlphaStar).
  • Modern alternative: Bayesian optimization + parallel runs.
#engineeringPermalink & quiz →

Interview: 'your DQN agent isn't learning — how do you debug?'

hard
  • (1) Verify env: hardcoded oracle policy should score well.
  • (2) Check reward signal: is it too sparse / too noisy?
  • (3) Value function sanity: init Q at 0, after N steps Q should reflect near-term reward magnitude.
  • (4) Exploration: is ε schedule too aggressive?
  • Are all actions being tried?
  • (5) Replay buffer: enough diversity?
  • (6) Target network: updating too fast / slow?
  • (7) Overestimation: try Double DQN.
  • (8) Gradient stability: check norms + reduce LR.
  • (9) Compare against baseline (random, SL BC).
#interview#engineeringPermalink & quiz →

Policy averaging — what is it and why?

medium
  • Take exponentially-weighted average of policy weights over training (Polyak averaging on policy).
  • Reduces variance of policy across training + gives smoother behavior.
  • Used in TD3, SAC.
  • Related concept: SWA (Stochastic Weight Averaging) — averaging final training weights improves generalization.
  • Modern: DreamerV3 averages world model + policy weights.
#engineeringPermalink & quiz →

How do you efficiently train RLHF at scale?

hard
  • (1) Sample generations offline, cache.
  • (2) Reward model inference batched separately from policy.
  • (3) Use PagedAttention / vLLM for fast rollout inference.
  • (4) DeepSpeed-Chat / trlX / OpenRLHF frameworks handle mixed policy-value training.
  • (5) Reference model on CPU or shared across GPUs.
  • (6) Reward normalization + advantage whitening per-batch.
  • (7) Save intermediate checkpoints — training is fragile.
  • Modern practice: careful engineering matters more than algorithm.
#llm#engineeringPermalink & quiz →

Your policy is perfect in simulation and useless on the real robot. How do you close the gap?

hard
  • Assume the policy has learned to exploit simulator inaccuracies, because that is the usual cause rather than a lack of capacity.
  • Domain randomization is the main tool: vary masses, friction, latencies, sensor noise and lighting during training so the policy must be robust across a distribution of dynamics rather than optimal for one wrong set.
  • Add realistic actuation delay and observation noise, which simulators tend to omit and which change control problems qualitatively.
  • Identify parameters from real measurements where you can, narrowing the randomization around reality.
  • Then fine-tune on a small amount of real data, and design the observation space to use signals whose simulated version you trust, since a policy that depends on an unrealistic sensor cannot transfer.
#applications#engineeringPermalink & quiz →

Two RL algorithms report different scores in a paper. Why should you be skeptical?

medium
  • Because reinforcement learning results have unusually high variance across seeds, and a difference computed from three runs is frequently noise.
  • The distribution of final performance is often bimodal, with some seeds failing entirely, so a mean hides more than it shows and a maximum over seeds is not a result at all.
  • Implementation details matter as much as the algorithm: observation normalization, advantage normalization, reward scaling and network initialization can move scores more than the contribution being claimed.
  • Evaluation protocol also differs, in the number of episodes, whether the policy is stochastic or greedy, and whether scores come from training or separate rollouts.
  • Ask for many seeds, confidence intervals or performance profiles, and identical tuning budgets for both methods.
#evaluation#engineeringPermalink & quiz →

Practise Reinforcement Learning

All 12 sectionsFlashcardsQuiz