How do you choose batch size for inference?
mediumAnswer
- Trade-off: larger batch → higher throughput + higher latency.
- Constraints: (1) memory (KV cache scales with batch × seq).
- (2) latency SLA (p99 must fit).
- (3) GPU compute pattern (matmul kernels want ~64+ batch).
- Dynamic batching: aggregate requests over 5-50ms window until batch full or timeout.
- Modern LLM serving: continuous batching adjusts dynamically.
- Rule: profile QPS vs latency, pick knee of curve; often batch=8-32 sweet spot for online.
Check yourself — multiple choice
- Just one
- Larger batch = higher throughput + higher latency; memory + latency SLA + compute pattern constraints; dynamic batching aggregates 5-50ms window; profile QPS-vs-latency; batch 8-32 often sweet spot
- Random
- Not real
Batch: dynamic 5-50ms window; profile QPS/latency; 8-32 sweet spot.
#serving#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.