What is benchmark contamination and how do you detect it?
hardAnswer
- The training corpus contains verbatim (or near-verbatim) copies of benchmark questions — the model memorizes answers and inflates eval scores.
- Detection: (1) exact-string search for benchmark questions in the training set; (2) 'canary' strings — inject unique markers into curated benchmarks and check for regurgitation; (3) held-out contemporary benchmarks after model cutoff.
- Modern LLM eval reports increasingly cite contamination-decontaminated scores.
Check yourself — multiple choice
- Impossible to detect
- Benchmark strings leaked into training → exact-string / canary detection + fresh post-cutoff benchmarks
- Solved by higher temperature
- Doesn't affect scores
Contamination: exact-string / canary / post-cutoff eval; contamination-decontaminated scores are standard.
#pretraining#evaluation#data
Practise LLMs & GenAI
214 interview questions in this topic.