Reinforcement Learning · section 11 of 12

Applications & safety

11 interview questions on applications & safety, each answered in full. Free to read, no account needed.

AlphaGo — what made it different from prior Go engines?

medium

$Silver et al. 2016.$
$(1) Deep NN combining value and policy (previously hand-crafted).$
$(2) Trained via supervised learning from human games \to reinforcement learning via self-play.$
$(3) MCTS guided by NN policy prior + value estimate.$
$(4) Beat top human Lee Sedol 4-1 in 2016.$
$Predecessors relied on hand-crafted features + Monte Carlo rollouts.$
$AlphaZero later dropped the human-game pretraining, learned purely from self-play.$

hard

$No — AlphaFold uses supervised learning on protein structure database (PDB).$
$Loss: predicted vs true atomic coordinates.$
$However, related work uses RL for de novo protein design (RFdiffusion + reward from AlphaFold2 verifier is RL-adjacent — search over sequence space with structure-quality reward).$
$Interview trap: distinguish protein STRUCTURE prediction (SL) from protein DESIGN (RL / search).$

hard

$OpenAI 2019.$
$(1) PPO in simulation with Automatic Domain Randomization (ADR): progressively harder randomization of physics parameters.$
$(2) LSTM policy inferring physical parameters online.$
$(3) Kociemba's algorithm for cube solution planning; RL for physical manipulation.$
$(4) Sim-to-real via ADR made a single-hand manipulation robust to real-world variation.$
$Demonstrated sim2real at scale for high-dimensional dexterous manipulation.$

hard

$(1) Massively parallel simulation (IsaacGym / Isaac Sim, MuJoCo MJX, Genesis) — thousands of robots in parallel on one GPU.$
$(2) PPO on domain-randomized environment (friction, mass, latency, terrain).$
$(3) Curriculum from flat to rough terrain.$
$(4) Reward shaping: forward velocity + energy efficiency + smoothness.$
$(5) Deploy zero-shot to real robot (Rudin et al. 2022, Anymal, Spot).$
$Trained in hours; robust real-world walking without fine-tuning.$

medium

$Real applications: data center cooling (DeepMind 40% reduction), chip floorplan (DeepMind Nature 2021), advertising bidding, portfolio optimization, drug discovery (design molecules), catalyst design, tokamak plasma control (DeepMind + EPFL).$
$Challenges: (1) sample efficiency (real interactions expensive).$
$(2) Safety constraints.$
$(3) Distribution shift between training and deployment.$
$(4) Reward specification hard.$
$Mostly offline RL + simulation + careful validation.$

medium

$Contextual bandit: user context = state, item shown = action, click/watch = reward.$
$Long-term reward \to full RL for session-level or LTV optimization.$
$Challenges: (1) partial feedback (only chosen item observed).$
$(2) Non-stationarity (user preferences drift).$
$(3) Delayed rewards (retention).$
$(4) Off-policy evaluation critical (can't A/B every candidate).$
$Modern: Thompson sampling for exploration, contextual bandits for retrieval, RL for long-horizon (Netflix, YouTube, TikTok).$

hard

$(1) Portfolio optimization: policy allocates capital across assets.$
$(2) Market making: bid/ask spread policy.$
$(3) Options hedging: dynamic hedging via RL.$
$Challenges: (1) non-stationary markets.$
$(2) Extremely low signal-to-noise.$
$(3) Adversarial (other traders learn).$
$(4) Backtesting can't capture real market impact.$
$Modern practice: careful backtest + shadow trading + tiny position sizes early.$
$Most quantitative funds combine RL with classical control + supervised signals.$

hard

$(1) Warm-start with content-based recommendations for cold start.$
$(2) Contextual bandit with Thompson sampling: Beta posteriors on CTR give principled exploration.$
$(3) Epsilon-greedy on tail items for coverage.$
$(4) Diversify via MMR / determinantal point processes to expose users to variety.$
$(5) Off-policy correction: log propensity scores, use IPS or doubly-robust estimators.$
$(6) Bandit for retrieval + supervised for ranking (industry pattern).$
$Monitor exploration cost via holdouts.$

#interview#applicationsPermalink & quiz →

medium

$(1) Atari 57 (DQN-era, still Rainbow / R2D2 / Agent57).$
$(2) MuJoCo continuous control (Hopper, Walker, Ant, Humanoid) — SAC / PPO staple.$
$(\mathrm{dm}_{\mathrm{control}})$
$(4) D4RL for offline RL.$
$(5) ProcGen / MiniGrid for generalization.$
$(6) IsaacGym / Isaac Sim for parallel robotics.$
$(7) MineRL, NetHack for exploration + open-ended learning.$
$(8) SMAC for cooperative multi-agent.$

hard

$Brohan et al. 2023 (Google).$
$Fine-tune vision-language model (PaLI-X / PaLM-E) to output action tokens for robotics.$
$Trained on mix of internet VQA data + robot trajectories.$
$Enables (1) generalizing web-scale semantic knowledge to robots ('bring me the extinct animal' \to picks toy dinosaur).$
$(2) Emergent chain-of-thought for manipulation.$
$Foundation of VLA (Vision-Language-Action) models — RT-2, OpenVLA, Pi-0.$

medium

$Whenever the problem is really supervised prediction with a decision rule on top, which covers most business cases.$
$If the action does not change the future state of the world, you do not need sequential decision making: predict the outcome and optimize the decision analytically.$
$Reinforcement learning also needs either a fast, faithful simulator or the willingness to let a bad policy act on real users, and most organizations have neither.$
$Sparse rewards, long delays between action and outcome, and no ability to explore safely all push the difficulty up sharply.$
$The honest recommendation is usually a contextual bandit, which keeps the exploration benefit without credit assignment over time, or a predictive model plus an optimizer.$

#interview#applicationsPermalink & quiz →