What parallelism strategies are combined to train a 70B model?
hardAnswer
- (1) Tensor parallelism (TP=4-8): split each matmul across GPUs on the same node using NVLink.
- (2) Pipeline parallelism (PP=4-16): split layers across nodes, microbatch to hide bubbles.
- (3) Data parallelism (DP=N): replicate across pipeline stages, each processes a different batch shard.
- (4) ZeRO/FSDP: shard optimizer + gradients + parameters across DP replicas.
- Total: 3D parallelism.
- Add sequence parallelism for long context, expert parallelism for MoE.
- Frameworks: Megatron-DeepSpeed, TorchTitan.
Check yourself — multiple choice
- Only data parallelism
- 3D parallelism: tensor + pipeline + data (+ ZeRO/FSDP); add sequence-parallel for long ctx, expert-parallel for MoE
- Only tensor parallelism
- One GPU is enough
70B training: tensor + pipeline + data + FSDP; +sequence for long ctx, +expert for MoE.
#pretraining#architecture
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How do you prevent MoE routing collapse?
- Why is the softmax + cross-entropy at the LM head a training bottleneck?
- What is embedding tying and when do LLMs use it?
- What is the context window and what tricks extend it?
- What is a Mixture-of-Experts (MoE) LLM?
- Walk through how one training step of an autoregressive LLM works.