Continuous batching vs static batching.
hardAnswer
- Static batching: form batch of N requests, run to completion (wait for slowest), then next batch.
- Problem: mix of long + short sequences wastes GPU on completed sequences.
- Continuous batching: at each token step, remove completed sequences + add new pending sequences.
- GPU fully utilized. 2-4x throughput improvement.
- Requires flexible KV cache (see PagedAttention).
- Industry standard: vLLM, TGI, TensorRT-LLM.
Check yourself — multiple choice
- Same
- Static: wait for slowest in batch (waste GPU on completed); Continuous: at each step, remove complete + add new → GPU fully utilized + 2-4x throughput; needs flexible KV cache (paged)
- Random
- Not real
Continuous batching: swap sequences per step; 2-4x throughput.
#serving#llmops
Practise MLOps & Data Quality
215 interview questions in this topic.