What can you cache in a RAG pipeline?
mediumAnswer
- (1) Embeddings: cache query embeddings by exact query hash (short-lived, small hit rate but cheap).
- (2) Retrieval results: cache retrieved chunk IDs per query hash (good hit rate for repeated queries).
- (3) LLM answers: cache (query, retrieved chunks) → answer for identical inputs (moderate hit rate).
- (4) Prompt cache: cache KV state of the system prompt + few-shot examples across all requests (100% hit rate for stable prefixes; huge cost cut).
- Combine layers for 40-80% cost reduction.
Check yourself — multiple choice
- Cache nothing
- Embeddings, retrieval results, answers, KV prompt cache — layer them for 40-80% cost cut
- Only LLM outputs
- Impossible to cache
RAG caching: embed / retrieval / answer / KV prompt caches → big cost + latency wins.
#rag#production#cost
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How much does it cost to pretrain a modern LLM?
- How should max_tokens be set in production?
- What is prompt caching and when does it help?
- How do you use one prompt to extract multiple fields at once vs many separate prompts?
- Walk through the components of a production RAG pipeline.
- How do you handle the freshness problem in RAG (docs change constantly)?