What is gradient accumulation and when do you use it?
easyAnswer
- Run several forward-backward passes with small mini-batches without calling optimizer.step(); sum the gradients.
- Then apply one optimizer step as if you had used a large batch.
- Simulates a large effective batch on limited GPU memory without changing statistics much (BN stats still per-microbatch — so combine with GroupNorm or SyncBN if BN is used).
Check yourself — multiple choice
- It reduces gradient magnitude by half each step
- Accumulate gradients across several small forward-backward passes, then step once — simulates a large batch
- It replaces the optimizer
- Only works on CPUs
Gradient accumulation simulates large batches when memory is tight.
#training#batch-size#distributed
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is gradient noise scale (GNS) and how do you use it?
- How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?
- How does batch size affect training dynamics?
- What problem does LAMB solve?
- How should learning rate scale with batch size?
- What is LARS and when is it used?