What is chunked prefill and why is it useful?
hardAnswer
- Split a long prompt's prefill into chunks of a few hundred tokens and interleave them with decode steps of other requests in the same batch.
- Prevents a single long prompt from starving decode users (a 32k-token prefill can take seconds → other users see TTL spikes).
- Reduces P99 TTFT dramatically without hurting throughput.
- Enabled by continuous batching + Paged Attention.
- Standard in modern serving stacks (vLLM, SGLang, TensorRT-LLM).
Check yourself — multiple choice
- One-shot prefill only
- Split long prompt prefill into chunks interleaved with decode → prevents long prompts from starving others → better P99 TTFT
- Only for training
- Same as streaming
Chunked prefill: interleave long prompt processing with active decode → P99 latency win.
#inference#serving#batching
Practise LLMs & GenAI
214 interview questions in this topic.