Your model trains well at batch size 256 but degrades at batch size 4. Why might batch normalization be the culprit?
hardAnswer
- Batch norm estimates the mean and variance from the batch itself, so with four samples those statistics are extremely noisy.
- The noise acts as an uncontrolled regularizer during training and, worse, the running averages accumulated for inference no longer match what the layer saw, so train and eval behaviour diverge.
- It also couples examples within a batch, which breaks any assumption of per-sample independence.
- The practical fixes are to switch to group norm or layer norm, whose statistics are computed per sample and are therefore batch-size independent, or to use a normalization-free architecture with careful initialization.
- This is exactly why detection and segmentation models, which run tiny batches of large images, standardly use group norm.
Check yourself — multiple choice
- Batch norm is batch-size independent
- Batch statistics become extremely noisy, running averages stop matching training behaviour, and examples get coupled — group or layer norm fixes it because they normalize per sample
- Only the learning rate matters
- Small batches always fail
Batch norm's statistics degrade with batch size; per-sample normalizations do not.
#normalization#batch-size#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What does Batch Normalization do?
- What is gradient accumulation and when do you use it?
- How does batch size affect training dynamics?
- What is gradient noise scale (GNS) and how do you use it?
- Why does a model with BatchNorm behave differently at train time vs eval time?
- How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?