EasyDeepLearn
Deep Learning · section 3 of 19

Initialization & gradients

6 interview questions on initialization & gradients, each answered in full. Free to read, no account needed.

What are vanishing and exploding gradients, and how do you fix them?

medium
  • In deep networks, chain-rule products can shrink toward zero (vanishing) or blow up (exploding), stalling learning.
  • Fixes: better initializations (He for ReLU, Xavier for tanh), normalization (BatchNorm, LayerNorm), skip connections (ResNet), non-saturating activations (ReLU family), gradient clipping (for RNNs and large models), and appropriate learning rates.
#training#gradientsPermalink & quiz →

What is Xavier (Glorot) initialization and why?

medium
  • Sample weights from a distribution with variance 2  /  (fanin  +  fanout)2\; / \;(\mathrm{fan}_{\mathrm{in}}\; + \;\mathrm{fan}_{\mathrm{out}}) — designed so that activations and gradients have roughly the same variance layer to layer for tanh/sigmoid activations.
  • Prevents both vanishing and exploding signals at initialization.
  • Default in older frameworks; replaced by He init when ReLU became the standard.
#initialization#trainingPermalink & quiz →

Why does He initialization use Var⁡  =  2  /  fanin\operatorname{Var}\; = \;2\; / \;\mathrm{fan}_{\mathrm{in}} instead of Xavier's 2  /  (fanin  +  fanout)2\; / \;(\mathrm{fan}_{\mathrm{in}}\; + \;\mathrm{fan}_{\mathrm{out}})?

medium
  • ReLU zeros out half of its inputs on average, halving the effective variance of activations.
  • He init doubles the variance to compensate: Var⁡(W)  =  2  /  fanin\operatorname{Var}(W)\; = \;2\; / \;\mathrm{fan}_{\mathrm{in}}.
  • This keeps activation variance stable through ReLU layers and prevents signals from collapsing to zero.
  • Default for any modern ReLU / Leaky-ReLU / GELU network.
#initializationPermalink & quiz →

When is orthogonal weight initialization useful?

hard
  • Orthogonal init sets weight matrices to random orthogonal matrices (singular values all 1), which preserves the norm of the input under linear map.
  • Especially useful for very deep networks and RNNs where repeated multiplication amplifies or shrinks signals — orthogonal keeps things unit-norm.
  • Slightly better than He on very deep nets without normalization.
#initialization#rnnPermalink & quiz →

What causes exploding gradients and how do you handle them?

medium
  • Exploding gradients happen when the product of many derivatives with singular values > 1 grows exponentially.
  • Common in RNNs (repeated  multiplication  by  Whh)(\mathrm{repeated}\;\mathrm{multiplication}\;\mathrm{by}\;W_{\mathrm{hh}}) and very deep networks without normalization.
  • Signs: loss becomes NaN, weight norms blow up.
  • Fixes: gradient clipping (by  norm,  typical  max⁡norm  =  1−5)(\mathrm{by}\;\mathrm{norm}, \;\mathrm{typical}\;\operatorname{max}_{\mathrm{norm}}\; = \;1 - 5), spectral normalization, better initialization, residual connections, and lower learning rate.
#gradients#rnn#trainingPermalink & quiz →

Gradient clipping by norm vs by value — which do you prefer?

easy
  • Clip-by-norm: rescale the whole gradient vector so its L2 norm is at most max⁡norm\operatorname{max}_{\mathrm{norm}}.
  • Preserves direction, only shrinks magnitude.
  • Preferred default.
  • Clip-by-value: clamp each element to [-clip, clip].
  • Distorts the gradient direction — element-wise clipping can flip the relative magnitudes of components.
  • Use clip-by-norm for RNNs and large-model training; clip-by-value only for specific numerical stability issues.
#gradients#trainingPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz