How do you trade off throughput vs latency in LLM serving?
mediumAnswer
- Larger batch size → higher throughput (better GPU utilization) but higher latency per request (waiting for batchmates).
- Smaller batch → lower latency, worse throughput.
- Latency-oriented deployments (chat) use small batches + speculative decoding + FP8.
- Throughput-oriented (offline batch processing, embeddings) use huge batches + higher context length.
- Modern serving frameworks (vLLM, TGI) let you set and knobs.
- Measure both P50 and P99 latency + tokens/sec throughput.
Check yourself — multiple choice
- Only one axis
- Large batch → throughput ↑ / latency ↑; small batch → latency ↓ / throughput ↓; chat = small batch + speculative decoding
- Batch doesn't matter
- Same tradeoff for training
Throughput ↔ latency trade-off via batch size; chat uses small batch + speculative decoding.
#inference#serving#latency
Practise LLMs & GenAI
214 interview questions in this topic.