EasyDeepLearn
Reinforcement Learning · section 5 of 12

Model-based & planning

4 interview questions on model-based & planning, each answered in full. Free to read, no account needed.

Monte Carlo Tree Search — how it works in AlphaGo/MuZero.

hard
  • Iteratively build a tree of possible action sequences.
  • Each iteration: (1) Select — walk from root using UCB1 (or PUCT) balancing exploration + Q.
  • (2) Expand — add new child node.
  • (3) Evaluate — value estimate from neural net (AlphaZero) or rollout.
  • (4) Backup — propagate value up.
  • After N iterations, pick move with highest visit count.
  • UCB: a = argmax Q + c √(ln⁡  N  /  na)(\operatorname{ln}\;N\; / \;n_{a}).
  • Foundation of AlphaGo, AlphaZero, MuZero.
#planning#model-basedPermalink & quiz →

AlphaZero — key ideas.

hard
  • (1) MCTS as policy-improvement operator: search improves the raw NN prior.
  • (2) Self-play generates training data: play against your current model.
  • (3) Single neural net outputs BOTH policy prior and value estimate.
  • (4) Train NN to predict MCTS output (targets: MCTS action distribution + game outcome).
  • (5) No handcrafted features.
  • Iterate → superhuman Go, Chess, Shogi in 24h.
  • Modern extension: MuZero learns dynamics model too.
#planning#model-basedPermalink & quiz →

Dyna — early model-based RL framework.

medium
  • Sutton 1990: interleave (1) real interaction with env, (2) model updates from real data, (3) planning: multiple simulated Q-learning updates from the learned model.
  • Rebuilds Q from cheap simulated experience.
  • Cornerstone of model-based RL.
  • Modern deep versions: Dyna-Q with NN model, MBPO (Janner et al. 2019).
  • Advantage: model-based sample efficiency + model-free performance.
#model-based#planningPermalink & quiz →

Model Predictive Control (MPC) in RL — how does it work?

medium
  • At each step: (1) roll out learned or hand-crafted model for horizon H, (2) optimize action sequence over that horizon (CEM, LQR, iLQR, gradient descent), (3) execute the first action, (4) re-plan next step.
  • Uses: robotics locomotion (PETS, PlaNet), industrial control, quadcopter flight.
  • Advantage: adapts to model errors via constant re-planning.
  • Weakness: computation per step.
  • Combined with learned model for MBRL.
#model-based#planningPermalink & quiz →

Practise Reinforcement Learning

All 12 sectionsFlashcardsQuiz