What is continuous / dynamic batching in LLM serving?
hardAnswer
- Naive static batching: batch of N requests, wait for the slowest to finish, then start the next batch → many GPU cycles wasted.
- Continuous batching (Yu 2022, vLLM): as soon as any request in the batch finishes, remove it and slot in a new request at that free position — GPU stays saturated.
- Combined with paged attention (KV cache in fixed pages), enables mixing requests of different lengths seamlessly.
- Typical throughput gain: 2-10x over static batching.
- Standard in vLLM, TGI, TensorRT-LLM.
Check yourself — multiple choice
- Static batching only
- Slot new requests into freed batch positions as others finish → keep GPU saturated → 2-10× throughput vs static batching
- One request at a time
- Same as beam search
Continuous batching: swap finished requests out mid-batch → 2-10× throughput.
#inference#serving#batching
Practise LLMs & GenAI
214 interview questions in this topic.