How does batch size affect training dynamics?
mediumAnswer
- Larger batch → lower gradient noise, closer to true gradient, allows higher learning rate (roughly LR ∝ sqrt(batch)).
- But too large batches tend to converge to sharper minima that generalize slightly worse (the 'generalization gap').
- Small batches are noisier — regularizing effect that often helps final accuracy.
- Rule: scale LR with batch and use warmup to keep training stable at very large batches (LARS/LAMB for extreme scales).
Check yourself — multiple choice
- Batch size has no effect on generalization
- Larger batch = less noise, higher LR allowed, but tends toward sharper minima and slightly worse generalization
- Smaller batch is always faster
- Batch size only affects memory
Batch controls noise ⇒ scale LR + warmup; watch generalization gap at extreme batch.
#training#batch-size#learning-rate
Practise Deep Learning
214 interview questions in this topic.