What is a golden test set and how do you maintain it?

medium

Answer

$Curated, high-quality, hand-labeled evaluation set that remains fixed across model versions — enables consistent comparison.$
$Properties: (1) representative of production distribution.$
$(2) covers edge cases + failure modes.$
$(3) contains fairness / bias slices.$
$(4) labeled by experts (not crowdsourced).$
$(5) NEVER used for training (leakage risk).$
$Maintenance: augment with new edge cases from production failures; periodic review + relabeling; freeze snapshot for each release.$

Check yourself — multiple choice

Random
Curated hand-labeled eval set fixed across versions; representative + edge cases + fairness slices + expert-labeled + never in training; augment with failures + freeze snapshots
Same as val
Not real

Golden set: fixed, expert-labeled, never in training; augment with failures.

#monitoring#data-quality

215 interview questions in this topic.