What is gradient checkpointing and its trade-off?
mediumAnswer
- Store activations at only a few 'checkpoint' layers instead of every layer.
- During backward, recompute the missing activations from the nearest checkpoint on demand.
- Memory drops from O(L) to O(sqrt(L)) with optimal checkpoints.
- Trade-off: ~30% more compute in the backward pass.
- Standard technique to fit larger models on limited GPUs (used in training large transformers).
Check yourself — multiple choice
- No memory savings
- Store activations at checkpoints, recompute others in backward → memory O(√L), +30% compute
- Doubles memory
- Only for CNNs
Gradient checkpointing: memory O(√L) at ~+30% compute cost.
#gradient-checkpointing#distributed
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does activation checkpointing interact with gradient accumulation and FSDP?
- Why does the backward pass require more memory than the forward pass?
- What is gradient accumulation and when do you use it?
- What problem does LAMB solve?
- Why does Adafactor use less memory than Adam?
- How should learning rate scale with batch size?