EasyDeepLearn

Different notions of return — MC, TD(0), n-step, GAE.

hard

Answer

  • MC return: Gt  =  ΣG_{t}\; = \;{\Sigma} γk{\gamma}^{k} rt+kr_{t + k} — full episode, unbiased, high variance.
  • TD(0): rt  +  γr_{t}\; + \;{\gamma} V(st+1)V(s_{t + 1}) — biased (bootstrap), low variance. n-step: rt  +  γr_{t}\; + \;{\gamma} rt+1r_{t + 1} + ... + γn−1{\gamma}^{n - 1} rt+n−1  +  γnr_{t + n - 1}\; + \;{\gamma}^{n} V(st+n)V(s_{t + n}) — knob n interpolates.
  • GAE (Generalized Advantage Estimation): weighted sum over n-step advantages with parameter λ — smooth bias-variance tradeoff.
  • GAE with λ ≈ 0.95 is default in PPO.
Check yourself — multiple choice
  • Same thing
  • MC (unbiased, high var), TD(0) (biased, low var), n-step (knob n), GAE(λ) smooth interpolation — PPO uses GAE(λ ≈ 0.95)
  • Random
  • Only MC

MC vs TD(0) vs n-step vs GAE: bias-variance interpolation.

#theory#actor-critic

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions