EasyDeepLearn

Prioritized replay — mechanism.

hard

Answer

  • Sample transitions with probability pip_{i} ∝ |δi{\delta}_{i}|^α (high TD error = more surprising = more informative).
  • Correct sampling bias with importance weights wi  =  (1/N  ⋅  1/Pi)w_{i}\; = \;(1 / N\; \cdot \;1 / P_{i})^β applied to the loss.
  • Improves data efficiency 2-3× on Atari.
  • Standard extension of DQN.
  • Modern replay libraries (segment tree implementation) allow O(log n) prioritized sampling.
Check yourself — multiple choice
  • Uniform sample
  • Sample transitions with prob ∝ |TD error|^α + importance weight correction; 2-3× data efficiency on Atari; segment-tree for O(log n) sampling
  • Random
  • Not real

Prioritized replay: p ∝ |δ|^α + IS weight; boosts sample efficiency.

#value-methods#deep-rl

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions