How does activation checkpointing interact with gradient accumulation and FSDP?
hardAnswer
- Activation checkpointing (recompute activations in backward) trades ~30% extra compute for O(√L) memory.
- Gradient accumulation reduces optimizer-step frequency but doesn't reduce peak activation memory.
- FSDP shards parameters/gradients but activation memory stays the same.
- All three are complementary — big LLM training uses checkpointing + gradient accumulation + FSDP + mixed precision + flash attention simultaneously to fit models 10x larger than would otherwise be possible.
Check yourself — multiple choice
- They cannot be combined
- Complementary — grad checkpoint saves activation memory, grad accumulation saves batch memory, FSDP saves param/opt/state; combine everything for big LLMs
- Only one can be used
- Same technique
Big-LLM recipe: checkpointing + accumulation + FSDP + mixed precision + flash attention.
#gradient-checkpointing#distributed
Practise Deep Learning
214 interview questions in this topic.