EasyDeepLearn

Double DQN — what problem does it solve?

medium

Answer

  • Vanilla DQN target: max⁡a\operatorname{max}_{a} Qtarget(s,  a)Q_{\mathrm{target}}(s, \;a).
  • Same network selects AND evaluates max action → overestimation bias (noise favors overestimated actions).
  • Double DQN: use online network to select action, target network to evaluate.
  • Target  =  Qtarget(s,  argmax⁡a  Qonline(s,  a))\mathrm{Target}\; = \;Q_{\mathrm{target}}(s, \;\operatorname{argmax}_{a}\;Q_{\mathrm{online}}(s, \;a)).
  • Reduces overestimation, gives more stable learning.
  • Standard extension in modern DQN implementations.
Check yourself — multiple choice
  • Same as DQN
  • Vanilla max⁡a\operatorname{max}_{a} Q → overestimation bias from noise; Double DQN: online net selects argmax, target net evaluates → decouples selection/evaluation
  • Random
  • Not real

Double DQN: decouple selection and evaluation to reduce overestimation.

#value-methods#deep-rl

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions