EasyDeepLearn

Fitted Q-Iteration — batch off-policy Q-learning.

hard

Answer

  • Alternate: (1) collect / accumulate dataset D of (s, a, r, s') transitions, (2) fit Qk+1Q^{k + 1} to targets yi  =  ri  +  γy_{i}\; = \;r_{i}\; + \;{\gamma} max⁡a\operatorname{max}_{a} Qk(si,  a)Q^{k}(si, \;a) via supervised regression (any regressor: tree, NN).
  • Repeat.
  • Batch-mode, off-policy, works with existing logged data.
  • Foundation of many offline RL methods and NFQ (Neural Fitted Q).
  • DQN = FQI + online replay + target net.
Check yourself — multiple choice
  • Same as DQN
  • Batch offline Q-learning: fit Qk+1Q^{k + 1} to r + γ max⁡a\operatorname{max}_{a} Qk(s,  a)Q^{k}(s, \;a) via supervised regression; foundation of NFQ + DQN + offline RL methods
  • Random
  • Not real

FQI: batch off-policy Q-learning via supervised regression on TD targets.

#value-methods#offline-rl

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions