Count-based exploration for large state spaces.

hard

Answer

$r_{\mathrm{int}}\; = \;{\beta}$
$Naive count fails in continuous / high-dim state \to use pseudo-counts from density model (Bellemare et al. 2016) or hashing (Tang et al. 2017).$
$Encourages visiting rare states.$
$Foundation of hard-exploration deep RL.$
$Modern successor: RND (Random Network Distillation) — cheaper + effective.$

Check yourself — multiple choice

Random
$r_{\mathrm{int}}\; = \;{\beta}$ /√N(s) — reward for rare states; naive count fails in high-dim → pseudo-counts via density model or hashing; RND is modern efficient successor
Same as UCB
Not real

Count-based: $r_{\mathrm{int}}$ ∝ 1/√N(s); pseudo-counts / hashing / RND for scale.

#exploration#deep-rl

214 interview questions in this topic.