How do you improve sample efficiency in deep RL?
mediumAnswer
- (1) Replay buffer + off-policy learning (DQN, SAC).
- (2) Model-based (Dreamer, MBPO).
- (3) Higher update-to-data ratio (many gradient steps per env step).
- (4) Auxiliary losses on encoder (representation learning during RL — SPR, CURL).
- (5) Transfer from pre-trained representations (VC-1, R3M for vision robotics).
- (6) Regularize to expert demos (BC + RL).
- Robotics: (7) sim-to-real to leverage cheap simulation.
Check yourself — multiple choice
- Random
- Replay + off-policy (SAC) / MBRL (Dreamer, MBPO) / high update-to-data ratio / auxiliary encoder losses (SPR, CURL) / pre-trained visual features (VC-1, R3M) / BC-then-RL / sim-to-real
- Not possible
- Only more data
Sample efficiency: off-policy + MBRL + high UTD + aux losses + pretrained + BC + sim2real.
#deep-rl#engineering
Practise Reinforcement Learning
214 interview questions in this topic.