What's the difference between prefill and decode in LLM inference?
mediumAnswer
- Prefill: process the entire input prompt in one parallel forward pass — compute-bound (GPU tensor cores at 90%+ utilization), fast (thousands of tokens/sec/GPU).
- Produces the KV cache.
- Decode: generate output tokens one at a time — memory-bandwidth bound (reload weights + KV cache per step), slow (30-200 tokens/sec).
- This split matters for cost / SLA: TTFT (time-to-first-token) depends on prefill; throughput on decode.
- Different optimizations target each phase (chunked prefill, continuous batching for decode).
Check yourself — multiple choice
- Same phase
- Prefill: compute-bound, parallel over full prompt; Decode: memory-bound, one token at a time — different bottlenecks and optimizations
- Both memory-bound
- Prefill happens at training only
Prefill (parallel, compute-bound) ≠ Decode (sequential, memory-bound).
#inference#kv-cache
Practise LLMs & GenAI
214 interview questions in this topic.