Generalization in RL — why is it hard?
hardAnswer
- Deep RL overfits to training environments — Cobbe et al. 2019 showed Atari-trained agents fail on procedurally-generated variants.
- Root causes: (1) narrow training distribution.
- (2) Memorization of trajectory-specific features.
- Fixes: (1) diverse env procedurally (ProcGen benchmark).
- (2) Regularization (dropout, weight decay).
- (3) Data augmentation.
- (4) Domain randomization.
- (5) Learn invariant / causal features.
- (6) Meta-learning.
- Ongoing challenge.
Check yourself — multiple choice
- No problem
- Deep RL overfits training envs (Cobbe 2019 ProcGen); root cause narrow distribution + memorization; fix via procedural env + regularization + augmentation + DR + invariant features + meta-learning
- Random
- Same as SL
RL generalization: overfits envs; procedural / regularize / augment / DR / meta.
#theory#engineering
Practise Reinforcement Learning
214 interview questions in this topic.