What is prompt caching and when does it help?
hardAnswer
- Cache the KV attention state of a large prefix (system prompt, few-shot examples, retrieved context) and reuse it across many requests that share that prefix.
- On subsequent calls, prefill only the new suffix — huge savings on TTFT (10-100x) and cost (50-90%).
- Supported by Anthropic API , Google (implicit), OpenAI (implicit prefix cache), and open-source (vLLM automatic prefix caching).
- Essential for RAG apps and agent loops with stable system prompts.
Check yourself — multiple choice
- Cache the outputs
- Cache KV state of a shared prefix → prefill only new suffix → 10-100× faster TTFT + 50-90% cost cut on RAG / agents
- Same as embedding cache
- Only training
Prompt caching: reuse KV of a shared prefix → massive TTFT + cost savings on RAG / agents.
#production#inference#kv-cache#cost
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What does 'automatic prefix caching' do in vLLM?
- How do you pick a GPU for serving a 70B LLM?
- Your LLM feature costs too much per request. What levers do you pull, in order?
- What is the KV cache and why does it matter for LLM inference?
- What is Grouped-Query Attention (GQA) and why does it matter?
- How much does it cost to pretrain a modern LLM?