Why does tanh cause vanishing gradients in deep nets?
mediumAnswer
- tanh' ≤ 1, and is close to 1 only near x = 0.
- Deep nets keep multiplying such derivatives during backprop — the product shrinks toward 0.
- With He/Xavier init, activations at each layer land in the saturating region , where the derivative is near zero.
- Fix: use ReLU/GELU which don't saturate on the positive side, plus normalization to keep activations in the useful range.
Check yourself — multiple choice
- tanh has unbounded gradients
- tanh' ≤ 1 saturates near ±1 — repeated multiplication → gradient → 0 in deep nets
- tanh gradients grow exponentially
- tanh only vanishes in CNNs
tanh derivative ≤ 1 and saturates → repeated products vanish in deep nets.
#activations#gradients
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- What are vanishing and exploding gradients, and how do you fix them?
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?
- What is 'dying ReLU' and how do you fix it?
- What are SiLU (Swish) and GELU, and why do transformers prefer them over ReLU?
- What does temperature do in a softmax?