What does Batch Normalization do?
mediumAnswer
- For each mini-batch, BN normalizes activations to zero mean and unit variance per feature, then applies learned scale and shift.
- Effects: faster convergence, higher learning rates, mild regularization, and reduced internal covariate shift.
- Downsides: depends on batch size, awkward for small batches, and behaves differently in train vs eval mode.
- LayerNorm (per-example) is preferred in transformers and RNNs.
Check yourself — multiple choice
- BatchNorm normalizes across features per example
- BatchNorm normalizes across the batch per feature, with learned scale/shift
- BatchNorm requires no learnable parameters
- BatchNorm is preferred in transformers
BN normalizes per feature across the batch, then rescales.
#normalization#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why does a model with BatchNorm behave differently at train time vs eval time?
- Your model trains well at batch size 256 but degrades at batch size 4. Why might batch normalization be the culprit?
- What are vanishing and exploding gradients, and how do you fix them?
- When do you use LayerNorm instead of BatchNorm?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?