EasyDeepLearn

Retrace(λ) — safe off-policy Q updates.

hard

Answer

  • Munos et al. 2016.
  • Q-target: r + γ Σl{\Sigma}_{l} (γλ)l({\gamma}{\lambda})l Πi=1l{\Pi}_{i = 1}^{l} cic_{i} δt+l{\delta}_{t + l} with ci  =  λc_{i}\; = \;{\lambda} min⁡(1,  π/πb)\operatorname{min}(1, \;{\pi} / {\pi}_{b}).
  • Guaranteed convergence in off-policy setting + low variance from truncation.
  • Underlies ACER algorithm.
  • Elegant unification of importance sampling + eligibility traces.
  • Modern off-policy actor-critics still cite Retrace's design principles.
Check yourself — multiple choice
  • Random
  • Q-target r + γ Σ (γλ)l({\gamma}{\lambda})l Π cic_{i} δt+l{\delta}_{t + l} with ci  =  λc_{i}\; = \;{\lambda} min⁡(1,  π/πb)\operatorname{min}(1, \;{\pi} / {\pi}_{b}); safe off-policy convergence + low variance; foundation of ACER
  • Same as V-trace
  • Not real

Retrace: safe off-policy Q via truncated IS + trace; foundation of ACER.

#actor-critic#theory

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions