Do you need bias terms in every layer of a modern neural network?
mediumAnswer
- Often no.
- When a layer is followed by BatchNorm or LayerNorm with learned shift (beta), the bias is redundant — the norm's beta term absorbs it.
- Standard practice: linear + norm → set bias=False on the linear.
- Modern transformers frequently omit biases from Q/K/V projections and MLP layers when they use RMSNorm or LayerNorm — small memory / compute savings, no accuracy hit.
Check yourself — multiple choice
- Bias is always mandatory
- Bias is redundant when the layer is immediately followed by BatchNorm/LayerNorm with a learned shift
- Bias must always be zero
- Bias only matters for classification
Norm layers' β absorbs the bias — skip bias to save params.
#fundamentals#normalization
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- What does Batch Normalization do?
- When do you use LayerNorm instead of BatchNorm?
- In one sentence, what is backpropagation?
- State the universal approximation theorem in one sentence and its practical caveat.
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?