Why does the backward pass require more memory than the forward pass?
mediumAnswer
- Reverse-mode autodiff needs the intermediate activations from the forward pass to compute gradients (the chain rule uses them).
- So all activations are stored until backward finishes.
- Peak memory ≈ activations + gradients + optimizer state.
- Fixes for memory pressure: gradient checkpointing (recompute activations during backward instead of storing), mixed precision, model / ZeRO parallelism.
Check yourself — multiple choice
- Backward uses less memory than forward
- Backward needs forward activations stored — peak memory ≈ activations + gradients + optimizer state
- Memory is independent of the pass direction
- The forward pass stores gradients
Activations must persist until the backward pass consumes them — memory peak.
#backprop#gradient-checkpointing#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What are vanishing and exploding gradients, and how do you fix them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?
- Why do residual (skip) connections help training deep networks?
- In one sentence, what is backpropagation?