What do residual connections actually fix?
mediumAnswer
- They make optimization tractable at depth.
- Without them, a deep stack must learn an identity mapping through many nonlinear layers just to preserve information, which gradient descent does badly, and the observed symptom is that a deeper network trains to a worse training error than a shallower one.
- That is an optimization failure, not overfitting.
- A residual branch gives the gradient a path that reaches early layers with its magnitude roughly intact, so signal neither vanishes nor is distorted by the product of many Jacobians.
- The broader consequence is that the network can represent a shallow function easily and add depth only where it helps, which is why residual blocks are in essentially every modern architecture.
Check yourself — multiple choice
- They reduce parameter count
- They fix a depth optimization failure: the identity is easy to represent and gradients reach early layers with magnitude intact, so deeper stops meaning worse training error
- They act only as regularization
- They replace normalization
Residuals address degradation in training error at depth, an optimization problem rather than overfitting.
#architectures#gradients#fundamentals
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Depth vs width — which do you scale first?
- How do you compute the parameter count of a linear layer and a convolution?
- How do FLOPs scale for a forward pass through an MLP and a transformer?
- ReLU vs sigmoid vs GELU — when do you use each?
- What are vanishing and exploding gradients, and how do you fix them?
- Why do residual (skip) connections help training deep networks?