How do you evaluate LLM apps cheaply at scale?
mediumAnswer
- (1) Sampling: eval on a small random slice (100-500) of production traffic.
- (2) Prioritize hard examples: cluster traffic, sample from each cluster.
- (3) LLM-judge on the sample: 100-1000× cheaper than human.
- (4) Regression suite: fixed set of golden examples to run on every release.
- (5) Automated red-team: adversarial LLM generates jailbreak probes overnight.
- (6) Statistical significance: with 200 examples per treatment, you can detect ~5% quality differences.
- Don't try to eval everything — sample + regress.
Check yourself — multiple choice
- Full eval every time
- Sample + cluster-aware sampling + LLM judge + regression suite + adversarial red-team; ~200 examples detects 5% differences
- No eval possible
- Human eval every time
LLM app eval at scale: sample + judge + regression + adversarial; ~200 examples suffices.
#evaluation#production#cost
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How much does it cost to pretrain a modern LLM?
- How should max_tokens be set in production?
- What is prompt caching and when does it help?
- How do you use one prompt to extract multiple fields at once vs many separate prompts?
- What can you cache in a RAG pipeline?
- How do you pick a GPU for serving a 70B LLM?