How do you size a replay buffer?

medium

Answer

$Trade-off: bigger buffer \to more diverse data, older stale off-policy data.$
$Rule: enough to cover diverse experience but not so large that stale data hurts.$
$Atari DQN: 1M transitions.$
$10^{6} - 10^{7}$
$Off-policy sensitivity: SAC works with larger buffers; DDPG less so.$
$Prioritized replay adjusts distribution automatically.$
$Off-policy correction (V-trace, ReTrace) helps with staleness in on-policy-ish algorithms.$

Check yourself — multiple choice

Random
Bigger buffer = more diversity but more staleness; DQN-Atari 1M, continuous control $10^{6} - 10^{7}$ ; prioritized replay + off-policy correction (V-trace) mitigate staleness
Not needed
Only tiny

Replay size: diversity vs staleness; Atari 1M; continuous control up to $10^{7}$ .

#value-methods#engineering

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lesson Flashcards Take the quiz

Related questions