How does PyTorch DDP work under the hood?
mediumAnswer
- Each GPU has a full replica of the model + its own data slice.
- Forward and backward happen independently.
- When a bucket of gradients is computed (during backward), DDP does an all-reduce across all GPUs so every rank ends up with the averaged gradient.
- Optimizer.step is then identical across GPUs → weights stay in sync.
- Gradient bucketing overlaps communication with computation, hiding all-reduce latency.
Check yourself — multiple choice
- One GPU processes everything
- Each GPU replicates the model, computes gradient on its shard, then all-reduces gradients → synced weights
- Only forward on each GPU
- No communication
DDP: replicated model + all-reduce gradients → synchronized weights across GPUs.
#distributed#training
Practise Deep Learning
214 interview questions in this topic.