EasyDeepLearn
LLMs & GenAI · section 18 of 18

Production concerns

1 interview question on production concerns, each answered in full. Free to read, no account needed.

How do you estimate LLM API cost for a workload?

medium
  • Cost  =  inputtokensper\mathrm{Cost}\; = \;\mathrm{input}_{\mathrm{tokens}}\mathrm{per}_request × inputprice  +  outputtokensper\mathrm{input}_{\mathrm{price}}\; + \;\mathrm{output}_{\mathrm{tokens}}\mathrm{per}_request × outputprice\mathrm{output}_{\mathrm{price}} × QPS × seconds.
  • Input tokens are usually 3-5× cheaper than output.
  • Watch out for: (1) system prompt + few-shot bloating input tokens; (2) unbounded max⁡tokens\operatorname{max}_{\mathrm{tokens}} on user side; (3) failed requests still charged in some APIs; (4) function-call round trips (each call is a request).
  • Optimization levers: prompt caching (cache system prompt → discount on input), tighter max⁡tokens\operatorname{max}_{\mathrm{tokens}}, batching for embeddings.
#cost#productionPermalink & quiz →

Practise LLMs & GenAI

All 18 sectionsFlashcardsQuiz