How does Paged Attention work?

hard

Answer

$vLLM (Kwon 2023) allocates the KV cache in fixed-size 'pages' (e.g., 16 tokens each) via a page table — like OS virtual memory.$
$Benefits: (1) no fragmentation as sequences grow / shrink; (2) memory sharing across sequences with a common prefix (parallel sampling, beam search); (3) enables continuous batching with variable-length sequences.$
$Reduces KV-cache memory waste from ~60% (naive contiguous allocation) to ~4%.$
$Foundation of vLLM's throughput advantage.$

Check yourself — multiple choice

Contiguous KV allocation
KV cache in fixed-size pages via a page table → no fragmentation + prefix sharing + variable-length batching (vLLM)
Same as memory pooling in CUDA
Only for training

Paged Attention: OS-style paging for KV cache → tiny fragmentation + prefix sharing.

#inference#kv-cache#serving

214 interview questions in this topic.