Model parallel vs data parallel — when do you need each?

hard

Answer

$Data parallel: replicate the model, split the batch across GPUs — dominant approach up to ~10B params on a single node.$
$Model parallel: split the model itself across GPUs — needed when the model exceeds one GPU's memory.$
$Two flavors: tensor parallel (split each matrix multiply across GPUs, e.g., Megatron) and pipeline parallel (put different layers on different GPUs, with microbatching to keep them all busy).$
$Modern LLM training combines all three (3D parallelism).$

Check yourself — multiple choice

They're identical
Data-parallel: split batch; model-parallel: split model (tensor / pipeline); LLM training combines both (3D parallelism)
Only model-parallel matters
Only for tiny models

DP: split batch; MP: split model (tensor + pipeline); modern LLMs combine into 3D parallelism.

#distributed#training

214 interview questions in this topic.