What causes exploding gradients and how do you handle them?
mediumAnswer
- Exploding gradients happen when the product of many derivatives with singular values > 1 grows exponentially.
- Common in RNNs and very deep networks without normalization.
- Signs: loss becomes NaN, weight norms blow up.
- Fixes: gradient clipping , spectral normalization, better initialization, residual connections, and lower learning rate.
Check yourself — multiple choice
- Only affects final layer
- Gradients grow exponentially — clip by norm, use spectral norm, residuals, lower LR
- Solved automatically by Adam
- Doesn't happen with normalization
Clip gradient norm, use spectral norm / residuals / lower LR.
#gradients#rnn#training
Practise Deep Learning
214 interview questions in this topic.