Why does He initialization use $\operatorname{Var}\; = \;2\; / \;\mathrm{fan}_{\mathrm{in}}$ instead of Xavier's $2\; / \;(\mathrm{fan}_{\mathrm{in}}\; + \;\mathrm{fan}_{\mathrm{out}})$ ?

medium

Answer

$ReLU zeros out half of its inputs on average, halving the effective variance of activations.$
$\operatorname{Var}(W)\; = \;2\; / \;\mathrm{fan}_{\mathrm{in}}$
$This keeps activation variance stable through ReLU layers and prevents signals from collapsing to zero.$
$Default for any modern ReLU / Leaky-ReLU / GELU network.$

Check yourself — multiple choice

He init is for tanh activations
ReLU halves activation variance; He compensates with $\operatorname{Var}\; = \;2 / \mathrm{fan}_{\mathrm{in}}$
He init sets all weights to 1
He init requires no math

ReLU zeros half of activations ⇒ He doubles the init variance.

#initialization

214 interview questions in this topic.