Model bias in MBRL — the challenge.
hardAnswer
- Learned model has errors that compound over rollout → policy trained in imagined trajectories exploits model exploits.
- Solutions: (1) short rollouts (MBPO uses only 1-15 steps of imagination between real data), (2) ensemble of models for uncertainty (PETS: use variance to penalize plans), (3) careful data collection (COMBO, MOReL) that regularizes toward known regions, (4) uncertainty-aware planning (avoid uncertain trajectories).
Check yourself — multiple choice
- No issue
- Model errors compound → policy exploits model bugs; fix with short rollouts (MBPO 1-15 steps) + model ensembles for uncertainty (PETS) + conservative training (COMBO / MOReL)
- Random
- Same as GD
MBRL model bias: short rollouts + ensembles + conservative regularization.
#model-based#theory
Practise Reinforcement Learning
214 interview questions in this topic.