What are typical tokens/sec numbers for popular LLMs on H100?
mediumAnswer
- Rough per-GPU decode throughput on H100 (batch 1, FP8, vLLM): 7B ~200-300 tok/s, 13B ~120-180, 70B (tensor-parallel 2×H100) ~50-80.
- With continuous batching, aggregate throughput scales 5-10× (many concurrent requests).
- Numbers vary by prompt length, quantization, framework.
- Reference numbers help capacity planning: 1 M2 Ultra 60GB gets ~30 tok/s on 70B Q4; RTX 4090 ~50 tok/s on 13B Q4.
Check yourself — multiple choice
- 1000 tok/s for 70B
- H100: ~200-300 tok/s for 7B, ~120-180 for 13B, ~50-80 for 70B (2×TP, FP8); batching multiplies aggregate 5-10×
- Same across all models
- Only CPU speeds
H100 decode rough: 7B ~250, 13B ~150, 70B ~65 tok/s single-request; batching multiplies aggregate.
#inference#serving#production
Practise LLMs & GenAI
214 interview questions in this topic.