Why does a model with BatchNorm behave differently at train time vs eval time?
mediumAnswer
- Train: BN uses batch statistics (mean/var of the current mini-batch), providing regularization noise.
- Eval: uses stored running averages accumulated during training via EMA.
- If those running stats are out of sync with training (small batches, non-i.i.d. batches, incorrect running momentum), eval accuracy can drop dramatically.
- Common bug: fine-tuning with BN in train mode but small batches contaminates running stats.
Check yourself — multiple choice
- BN is identical in train and eval
- Train uses batch stats, eval uses running EMA — mismatch causes accuracy drops after fine-tuning
- Eval uses batch stats
- BN is disabled at inference
BN train ≠ eval: batch stats vs running EMA — a common source of bugs.
#normalization#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What does Batch Normalization do?
- Your model trains well at batch size 256 but degrades at batch size 4. Why might batch normalization be the culprit?
- What are vanishing and exploding gradients, and how do you fix them?
- When do you use LayerNorm instead of BatchNorm?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?