How do you estimate LLM API cost for a workload?
mediumAnswer
- _request × _request × × QPS × seconds.
- Input tokens are usually 3-5× cheaper than output.
- Watch out for: (1) system prompt + few-shot bloating input tokens; (2) unbounded on user side; (3) failed requests still charged in some APIs; (4) function-call round trips (each call is a request).
- Optimization levers: prompt caching (cache system prompt → discount on input), tighter , batching for embeddings.
Check yourself — multiple choice
- Free service
- × × × QPS; input often 3-5× cheaper; watch out for system
- Only per request
- Same as GPU cost
LLM cost: input + output tokens × price × QPS; optimize via caching / tight .
#cost#production
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How much does it cost to pretrain a modern LLM?
- How should max_tokens be set in production?
- What is prompt caching and when does it help?
- How do you use one prompt to extract multiple fields at once vs many separate prompts?
- What can you cache in a RAG pipeline?
- How do you pick a GPU for serving a 70B LLM?