Explain the three ZeRO stages.
hardAnswer
- ZeRO (Zero Redundancy Optimizer) partitions training state across GPUs to save memory.
- Stage 1: shard the optimizer state (Adam's m and v) across N GPUs → memory / N.
- Stage 2: also shard the gradients.
- Stage 3 (FSDP): also shard the parameters themselves — each GPU only holds a 1/N slice of the model at rest and gathers full weights only when needed.
- Enables training LLMs many times bigger than a single-GPU capacity.
Check yourself — multiple choice
- Only one stage exists
- Stage 1: shard optimizer state; Stage 2: also shard gradients; Stage 3: also shard parameters (FSDP) — memory ÷ N
- ZeRO removes parameters
- ZeRO doubles memory
ZeRO: shard optimizer / gradients / parameters across GPUs → memory ÷ .
#distributed#training
Practise Deep Learning
214 interview questions in this topic.