Tensor parallelism vs pipeline parallelism vs data parallelism.

hard

Answer

$Data parallelism (DDP / FSDP): same model replicated; different data per GPU; all-reduce gradients.$
$Tensor parallelism (Megatron): split large layers (attention, FFN) across GPUs; each does portion of matmul + all-reduce.$
$Pipeline parallelism (GPipe / PipeDream): different layers on different GPUs; micro-batches flow through pipeline.$
$Combine all three for LLM training (3D parallelism).$
$Modern: FSDP + TP + PP + expert parallelism (MoE).$
$ZeRO stages 1-3 optimize memory.$

Check yourself — multiple choice

Random
DP (DDP/FSDP): replicate model + all-reduce gradients; TP (Megatron): split large layers across GPUs; PP (GPipe/PipeDream): different layers on different GPUs + micro-batches; combine all = 3D parallelism; FSDP + TP + PP + expert (MoE)
Same
Not real

3D parallelism: Data + Tensor + Pipeline + Expert (MoE).

#infrastructure#advanced

215 interview questions in this topic.