How do you handle biased ground truth collection?

hard

Answer

$Feedback often depends on model output.$
$Recsys: only see clicks on shown items — no signal for unshown.$
$Fixes: (1) exploration: show random items occasionally for unbiased data.$
$(2) inverse propensity scoring (IPS): weight by 1/P(shown).$
$(3) counterfactual estimator.$
$(4) small holdout randomly sampled.$
$(5) explicit ratings vs implicit.$
$Interview red flag: candidate optimizes offline metrics on biased data + surprised when online is different.$

Check yourself — multiple choice

Random
Feedback depends on model output (recsys clicks) → biased; fix via exploration random items + IPS (weight by 1/P) + counterfactual + random holdout + explicit vs implicit; optimize on unbiased
Just ignore
Not real

Biased GT: exploration + IPS + counterfactual + random holdout.

#data-quality#monitoring

215 interview questions in this topic.