Accuracy has probably dropped but labels arrive 60 days late. What can you monitor now?

hard

Answer

$Everything upstream of the label.$
$Input drift per feature, using a population stability index or a Kolmogorov-Smirnov test against a fixed training reference, catches the changes most likely to matter.$
$Prediction drift is often the strongest single signal, because a shifted score distribution means either inputs changed or the model is behaving differently.$
$Add proxy outcomes available sooner than the real label, such as a click when the target is a 60-day conversion, and monitor their relationship to predictions.$
$Watch data quality separately, since null rates and cardinality changes precede most silent failures.$
$Then check for concept drift indirectly by comparing calibration on whatever labels do trickle in early, accepting that they are a biased sample.$

Check yourself — multiple choice

Nothing until labels arrive
Input drift per feature against a fixed reference, prediction score drift, data quality metrics, and early proxy outcomes — accepting that early labels are a biased sample
Only retrain on a schedule
Monitor server latency

Without labels, input and prediction drift plus data quality and proxies are the available signals.

#monitoring#data-quality

215 interview questions in this topic.