Why do residual (skip) connections help training deep networks?
easyAnswer
- They provide a shortcut so gradients can flow directly back to earlier layers, alleviating vanishing gradients.
- They also make it easy for a layer to learn the identity function, so adding depth can't hurt.
- Residual blocks let us train 100+ layer networks (ResNet) and are used in transformers too (skip around attention and MLP).
Check yourself — multiple choice
- Skip connections increase vanishing gradients
- Skip connections give gradients a shortcut and enable very deep training
- Skip connections replace activations
- Transformers do not use skip connections
Residuals ease optimization by providing a direct path for gradients.
#architectures#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is stochastic depth / DropPath and where is it used?
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?
- Depth vs width — which do you scale first?