Distributional RL (C51, QR-DQN) — the idea.

hard

Answer

$Model the FULL distribution of returns Z(s, a) instead of just its mean Q(s, a) = E[Z(s, a)].$
$C51 (Bellemare et al. 2017): discretize into 51 atoms, learn categorical distribution over returns.$
$QR-DQN: quantile regression over N quantiles.$
$IQN: implicit quantile network.$
$Benefits: richer signal, better risk-sensitivity, empirically better performance.$
$Z(s, \;\operatorname{argmax}_{a}\;Q)$

Check yourself — multiple choice

Same as DQN
Model full return distribution Z(s, a), not just mean; C51 = 51 atoms + categorical; QR-DQN quantile; IQN implicit — richer signal, risk-sensitive
Random
Not real

Distributional RL: learn Z(s, a) distribution; C51 / QR-DQN / IQN.

#value-methods#deep-rl

214 interview questions in this topic.