Which GPU for training vs inference?

medium

Answer

$Training: (1) NVIDIA A100 / H100 (data center): 80GB VRAM, NVLink, HBM3, best.$
$(2) H200 / B200 latest.$
$(3) TPU v4/v5 (Google) alternative.$
$(4) Rent from cloud (AWS p4/p5, GCP TPU, Lambda Labs, CoreWeave).$
$Inference: (1) L40S / A10G (data center inference-optimized, cheaper).$
$(2) T4 (older, budget).$
$(3) L4 for video / low-power.$
$(4) Consumer 4090 / 5090 for small teams (48GB via 4090 pairs).$
$LLM inference: 7B fits on 1x A10G, 70B needs 2x A100 or 4x A10G with quant.$

Check yourself — multiple choice

Same
Training: A100/H100 (80GB+NVLink+HBM3) / H200/B200 / TPU v4-v5 / cloud rent; Inference: L40S/A10G/T4/L4/consumer 4090; 7B on 1x A10G, 70B on 2x A100 or 4x A10G quant
Random
Not real

GPU: H100/A100 train / L40S/A10G inference; size fits VRAM.

#infrastructure

215 interview questions in this topic.