EasyDeepLearn

C51 — how does the projected Bellman update work?

hard

Answer

  • Distribution over N=51 atoms ziz_{i} in [Vmin,  Vmax][V_{\mathrm{min}}, \;V_{\mathrm{max}}] with probability pi(s,  a)p_{i}(s, \;a).
  • Bellman: sample transition, compute target atom ziz_{i}' = clip(r  +  γ  zi,  Vmin,  Vmax)\mathrm{clip}(r\; + \;{\gamma}\;z_{i}, \;V_{\mathrm{min}}, \;V_{\mathrm{max}}); this may not land on a valid atom → distribute probability mass proportionally to two nearest atoms (linear interpolation).
  • Loss: KL divergence between projected target distribution and current Q distribution.
  • Elegant, richer signal than Q mean-value.
Check yourself — multiple choice
  • Random
  • 51 atoms in [Vmin,  Vmax][V_{\mathrm{min}}, \;V_{\mathrm{max}}]; project Bellman target r + γz to nearest two atoms (linear interp); KL loss between projected target and current distribution
  • Same as DQN
  • Not real

C51: 51 atoms + projected Bellman + KL loss.

#value-methods#deep-rl

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions