How do you build a good golden eval set?
mediumAnswer
- (1) Diverse: cover intended tasks (chat, code, extraction), personas, languages, and edge cases (jailbreaks, PII, tricky inputs).
- (2) Labeled: humans provide expected outputs or acceptance criteria.
- (3) Realistic: draw from production traffic (with PII scrubbed) — synthetic data is a poor proxy.
- (4) Small (100-1000) but re-usable across releases.
- (5) Rotated: retire memorized examples, add new failures found in production.
- (6) Metrics: exact match where possible, LLM-judge with rubric for open-ended, calibrated to humans.
- Best investment for any serious LLM team.
Check yourself — multiple choice
- Random synthetic examples
- Diverse + human-labeled + realistic (production-drawn) + small reusable + rotated + calibrated metrics — best eval investment
- Only public benchmarks
- Only user feedback
Golden set: diverse + labeled + realistic + small + rotated + calibrated → highest-leverage eval.
#evaluation#production
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What LLM observability tools do you deploy?
- How do you evaluate LLM apps cheaply at scale?
- You have no evaluation set for a new LLM feature. How do you build one quickly?
- How do you evaluate an LLM system?
- How much does it cost to pretrain a modern LLM?
- What is benchmark contamination and how do you detect it?