Your LLM feature costs too much per request. What levers do you pull, in order?
mediumAnswer
- Start with the cheapest wins.
- Cache: identical and near-identical requests are common, and a semantic cache can remove a large share of traffic.
- Shorten the prompt, since few-shot examples and boilerplate often dominate the token count, and prompt caching makes a stable prefix nearly free.
- Route by difficulty: send easy requests to a small model and escalate only when a cheap check says the answer is uncertain.
- Then trim output length, which usually costs more per token than input.
- Only after that consider distilling a small model on your own traffic, or self-hosting a quantized model, since both add real operational burden.
Check yourself — multiple choice
- Immediately self-host
- Cache, shorten prompts and use prefix caching, route by difficulty, trim output length, then consider distillation or self-hosting
- Only reduce quality
- Cost cannot be reduced
Caching, prompt size, routing and output length come before self-hosting.
#cost#production#inference
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is prompt caching and when does it help?
- How do you pick a GPU for serving a 70B LLM?
- How much does it cost to pretrain a modern LLM?
- How should max_tokens be set in production?
- How do you use one prompt to extract multiple fields at once vs many separate prompts?
- What can you cache in a RAG pipeline?