How do you pick a GPU for serving a 70B LLM?
hardAnswer
- 70B in FP16 = 140 GB VRAM for weights alone → doesn't fit on a single A100 80GB or H100 80GB.
- Options: (1) tensor-parallel across 2 GPUs (2× H100/A100); (2) 4-bit quantize to ~35GB → fits on 1× H100 80GB with room for KV cache.
- Per-token throughput: A100 ~30-40 tokens/sec, H100 ~50-70 tokens/sec (dense), H200 ~100+.
- For latency-sensitive apps: H100/H200 (higher memory bandwidth).
- For throughput: batch on H100 or A100.
- Consumer options (RTX 4090 24GB) can run 7B-13B quantized but not 70B.
Check yourself — multiple choice
- Any GPU works
- 70B FP16 = 140GB → tensor-parallel 2×80GB or quantize to 4-bit for single H100; H100/H200 for latency, A100 for cost
- One RTX 4090
- Only A100
70B: 2×80GB TP or 4-bit on 1× H100; H100/H200 for latency; A100 for cost.
#inference#serving#cost#production
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is prompt caching and when does it help?
- vLLM vs TensorRT-LLM vs TGI vs SGLang — how do you pick?
- What does 'automatic prefix caching' do in vLLM?
- What role does NVIDIA Triton play in LLM serving?
- What are typical tokens/sec numbers for popular LLMs on H100?
- Why is cold-starting an LLM slow and how do you fix it?