You have no evaluation set for a new LLM feature. How do you build one quickly?
mediumAnswer
- Start from real traffic or realistic drafts rather than inventing questions.
- Collect 100 to 200 representative inputs, deliberately oversampling the hard and weird cases, since uniform sampling under-represents exactly what breaks.
- Write the expected behaviour for each, as a reference answer where one exists or as a checklist of things the answer must contain.
- Freeze the set and version it.
- Add regression cases every time a bug is reported, so the set grows into the shape of your actual failure modes.
- A couple of hundred well-chosen cases catches far more than a public benchmark that does not resemble your task.
Check yourself — multiple choice
- Use a public benchmark only
- Curate 100-200 real inputs oversampling hard cases, define expected behaviour, freeze and version it, and add every reported bug as a regression case
- Skip evaluation
- Generate everything synthetically
A small, versioned, traffic-derived set with regression cases beats a generic benchmark.
#evaluation#production
Practise LLMs & GenAI
214 interview questions in this topic.