How is FSDP different from DDP?
hardAnswer
- PyTorch FSDP (Fully Sharded Data Parallel) implements ZeRO-3: each GPU stores only a shard of the parameters, gradients, and optimizer state.
- When a layer runs forward, FSDP all-gathers its full parameters, computes, then discards them (or shards again).
- Enables 10-100x larger models than DDP on the same hardware, at the cost of extra communication.
- Modern default for training large models in PyTorch.
Check yourself — multiple choice
- Same as DDP
- Shards params/gradients/optimizer, all-gathers layers on demand → much larger models, more comms
- Only for CPU
- FSDP disables gradients
FSDP (ZeRO-3): shard params + on-demand all-gather → train big models on modest hardware.
#distributed#training
Practise Deep Learning
214 interview questions in this topic.