How do you optimize LLM inference cost?

medium

Answer

$(1) Choose right model tier: smaller model (Haiku, Mini) for easy tasks, only escalate hard.$
$(2) Prompt caching (Anthropic / OpenAI): 30-90% off shared prefixes.$
$(3) Batching where latency-tolerant.$
$(4) Distill to smaller model for specific tasks.$
$(5) Quantize (int4 / FP8) self-hosted.$
$(6) Truncate context to essential.$
$(7) Cache responses for deterministic queries.$
$(8) Use open-source (Llama, Mistral) for privacy + cost.$
$(9) Fine-tune smaller open-source to beat closed on your task.$
$Monitor $/query.$

Check yourself — multiple choice

Just cheaper
Right tier (small first, escalate) + prompt cache (30-90% off) + batching + distill + quant + truncate + response cache + open-source + fine-tune small; monitor $/query
Random
Not real

LLM cost: tier + cache + batch + distill + quant + truncate + OSS + fine-tune.

#llmops

215 interview questions in this topic.