What are vanishing and exploding gradients, and how do you fix them?
mediumAnswer
- In deep networks, chain-rule products can shrink toward zero (vanishing) or blow up (exploding), stalling learning.
- Fixes: better initializations (He for ReLU, Xavier for tanh), normalization (BatchNorm, LayerNorm), skip connections (ResNet), non-saturating activations (ReLU family), gradient clipping (for RNNs and large models), and appropriate learning rates.
Check yourself — multiple choice
- Vanishing gradients only affect CNNs
- Skip connections and better initialization help mitigate vanishing gradients
- Gradient clipping causes exploding gradients
- Sigmoid activations prevent vanishing gradients
Residual connections and He/Xavier init are the standard defenses.
#training#gradients
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What causes exploding gradients and how do you handle them?
- Gradient clipping by norm vs by value — which do you prefer?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?
- Why do residual (skip) connections help training deep networks?