Why does stacking linear layers without nonlinearity collapse to a single linear layer?
easyAnswer
- The composition of linear maps is a linear map: .
- No matter how many layers you stack, the effective function is W x + b.
- Non-linear activations (ReLU, GELU, tanh) break this and unlock representational power that grows with depth.
Check yourself — multiple choice
- Stacked linear layers approximate any function
- Linear composed with linear is still linear; nonlinearity gives depth its power
- ReLU is a linear function
- Bias terms alone break linearity
Composition of linear maps is linear — you need a nonlinearity between layers.
#fundamentals#activations
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- What does temperature do in a softmax?
- In one sentence, what is backpropagation?
- State the universal approximation theorem in one sentence and its practical caveat.
- Depth vs width — which do you scale first?
- Why can't you initialize a neural net with all zeros?