Depth vs width — which do you scale first?
mediumAnswer
- Depth compounds representational capacity exponentially in the number of piecewise-linear regions for ReLU networks, while width only grows it polynomially.
- So depth is usually the better initial investment for representation.
- However, very deep networks are harder to optimize — need normalization, residual connections, and careful init.
- Modern architectures (transformers, ConvNeXt) balance both via 'compound scaling'.
Check yourself — multiple choice
- Width is exponentially more powerful than depth
- Depth compounds capacity exponentially but needs residuals + normalization to train
- Depth and width are equivalent
- Only width matters for CNNs
Depth grows expressiveness exponentially — invest there first, then balance.
#fundamentals#architectures
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How do you compute the parameter count of a linear layer and a convolution?
- How do FLOPs scale for a forward pass through an MLP and a transformer?
- What do residual connections actually fix?
- ReLU vs sigmoid vs GELU — when do you use each?
- Why do residual (skip) connections help training deep networks?
- In one sentence, what is backpropagation?