How do you evaluate an LLM system?
mediumAnswer
- Use a layered eval: (1) automatic benchmarks (MMLU, HumanEval, GSM8K) for capabilities; (2) task-specific evals with reference answers (exact match, ROUGE, F1); (3) LLM-as-a-judge for rubric-based scoring — validate against human raters; (4) human eval on a small held-out set for critical behaviors; (5) production monitoring (feedback, hallucination rate, latency, cost, safety flags).
Check yourself — multiple choice
- MMLU alone is enough to trust a production LLM
- LLM-as-a-judge must be validated against human judgments
- Task-specific evals are unnecessary
- Production monitoring is not part of evaluation
LLM judges are useful but biased — always calibrate against humans.
#evaluation#benchmarks
Practise LLMs & GenAI
214 interview questions in this topic.