What does 'automatic prefix caching' do in vLLM?
hardAnswer
- vLLM hashes the token prefix of each request.
- When two requests share a prefix (same system prompt + few-shot examples), the second one reuses the first's KV cache for that prefix — only its unique suffix needs prefill.
- Massive speedup for RAG / chat where system prompts + retrieved context are shared.
- Works transparently; users don't need to declare a cache key.
- Hit rate depends on prefix stability; agent loops with stable system prompts see 60-90% cache hit.
Check yourself — multiple choice
- Users declare cache keys
- Transparent hash-based reuse of the KV cache when requests share a prefix — huge win for RAG and stable system prompts
- Only in TensorRT
- Same as speculative decoding
vLLM prefix cache: hash-based transparent KV reuse for shared prefixes.
#inference#kv-cache#serving#cost
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is prompt caching and when does it help?
- How does Paged Attention work?
- How do you pick a GPU for serving a 70B LLM?
- What is the KV cache and why does it matter for LLM inference?
- What is Grouped-Query Attention (GQA) and why does it matter?
- What's the difference between prefill and decode in LLM inference?