EasyDeepLearn
Deep Learning · section 17 of 19

Efficiency & distributed training

2 interview questions on efficiency & distributed training, each answered in full. Free to read, no account needed.

What is gradient checkpointing and its trade-off?

medium
  • Store activations at only a few 'checkpoint' layers instead of every layer.
  • During backward, recompute the missing activations from the nearest checkpoint on demand.
  • Memory drops from O(L) to O(sqrt(L)) with optimal checkpoints.
  • Trade-off: ~30% more compute in the backward pass.
  • Standard technique to fit larger models on limited GPUs (used in training large transformers).
#gradient-checkpointing#distributedPermalink & quiz →

How does activation checkpointing interact with gradient accumulation and FSDP?

hard
  • Activation checkpointing (recompute activations in backward) trades ~30% extra compute for O(√L) memory.
  • Gradient accumulation reduces optimizer-step frequency but doesn't reduce peak activation memory.
  • FSDP shards parameters/gradients but activation memory stays the same.
  • All three are complementary — big LLM training uses checkpointing + gradient accumulation + FSDP + mixed precision + flash attention simultaneously to fit models 10x larger than would otherwise be possible.
#gradient-checkpointing#distributedPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz