How should learning rate scale with batch size?
mediumAnswer
- Linear rule (Goyal et al., 2017): .
- Works up to ~8k batch in vision with sufficient warmup.
- Sqrt rule: LR ∝ sqrt(batch) — more conservative, sometimes better for Adam / transformers.
- Very large batches (>32k) need layer-wise scaling (LARS, LAMB).
- Always combine with warmup: large batches need longer warmup to avoid divergence.
Check yourself — multiple choice
- LR is independent of batch size
- Linear rule LR ∝ batch (with warmup); sqrt for Adam; LARS/LAMB above 32k batch
- LR should shrink with batch size
- Batch size only affects loss
Linear LR-batch scaling + warmup up to ~8k; sqrt for Adam; LARS/LAMB for extreme scales.
#batch-size#learning-rate#distributed
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is gradient accumulation and when do you use it?
- How does batch size affect training dynamics?
- What problem does LAMB solve?
- What is gradient noise scale (GNS) and how do you use it?
- What is LARS and when is it used?
- How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?