IMPALA — how does it scale actor-critic?
hardAnswer
- Actor-learner architecture: many parallel actor threads roll out trajectories using slightly-stale policy, single learner updates.
- Off-policy correction via V-trace (Espeholt et al. 2018): truncated importance sampling with clipping.
- Enables massive scaling (thousands of actors).
- Foundation of distributed deep RL (used in AlphaStar, OpenAI Five).
- Trades on-policyness for throughput + parallel scaling.
Check yourself — multiple choice
- Same as A3C
- Actor-learner arch: many stale-policy actors + single learner; V-trace off-policy correction (truncated IS + clipping); scales to 1000s of actors — AlphaStar / OpenAI Five
- Random
- Not real
IMPALA: actor-learner + V-trace; distributed deep RL scale.
#actor-critic#deep-rl#engineering
Practise Reinforcement Learning
214 interview questions in this topic.