The 'noisy TV problem' in curiosity-driven exploration.
hardAnswer
- If agent gets intrinsic reward for high prediction error, it can find sources of pure noise (TV showing static, random-generated stimuli) and stay 'exploring' those forever — infinite curiosity but no learning.
- Fixes: (1) inverse-model features (ICM) — only rewards prediction errors about controllable aspects.
- (2) RND — errors on RANDOM target eventually shrink, unlike true entropy.
- Classic argument for control-relevant curiosity.
Check yourself — multiple choice
- Random
- Curiosity on prediction error can be exploited by pure-noise sources ('noisy TV') → infinite curiosity, no learning; fix with ICM inverse features or RND (predictor eventually catches up)
- Not real
- Same as bias
Noisy TV: curiosity fooled by pure noise; fix ICM inverse features / RND.
#exploration
Practise Reinforcement Learning
214 interview questions in this topic.