How do you cache LLM prompts effectively?
hardAnswer
- (1) KV cache reuse (prefix caching): shared system prompt cached across users; new requests reuse. vLLM / Anthropic support natively.
- Saves 30-90% compute for shared prefixes.
- (2) Result cache (memoize): hash prompt → cached response.
- Works for deterministic prompts.
- (3) Embedding cache: dedupe similar prompts via vector similarity + return cached.
- (4) Batch level: within batch, common prefix computed once.
- Modern: Anthropic 'prompt caching' + Gemini 'context caching' + OpenAI 'prompt caching' (auto).
Check yourself — multiple choice
- Random
- Prefix caching (shared system prompt reuse, vLLM/Anthropic native, 30-90% save) + result cache (hash→response) + embedding cache (vector similar) + batch prefix once; Anthropic/Gemini/OpenAI prompt caching
- No cache
- Not real
LLM cache: prefix + result + embedding + batch; 30-90% save on shared prefix.
#serving#llmops
Practise MLOps & Data Quality
215 interview questions in this topic.