How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?

medium

Answer

$You run several smaller micro-batches, summing or averaging their gradients, and only step the optimizer once at the end.$
$Mathematically the update matches a single large batch, so the loss curve is nearly identical while peak activation memory stays at the micro-batch level.$
$What it does not fix is time: you still do the same amount of computation, so the step takes proportionally longer, and it does not give the throughput benefit of a genuinely larger batch on more hardware.$
$It also interacts badly with batch normalization, whose statistics are computed per micro-batch and therefore reflect the small size, not the effective one.$
$Remember to scale the loss so accumulation averages rather than sums.$

Check yourself — multiple choice

It makes training faster
Micro-batches accumulate gradients for one optimizer step, matching a large batch's update with small activation memory — but not its speed, and batch norm statistics still see the micro-batch
It reduces total computation
It changes the loss function

Accumulation buys effective batch size in memory terms only; time cost and batch-norm statistics are unchanged.

#distributed#training#batch-size

214 interview questions in this topic.