What is model decay and how do you measure it?

medium

Answer

$Gradual degradation of model performance over time due to drift + concept change + world evolving.$
$Measure: (1) rolling accuracy on freshly-labeled batches.$
$(2) accuracy vs training vintage.$
$(3) compare snapshot models: retrain monthly, evaluate on latest data — degradation curve.$
$(4) survival analysis on 'time until performance drops X%'.$
$Decay rate informs retraining cadence.$
$Some models decay in days (fraud), others in years (image classifiers on stable domains).$

Check yourself — multiple choice

Random
Rolling accuracy on fresh batches + accuracy vs training vintage + monthly-snapshot degradation curve + survival analysis 'time to X% drop'; rate informs retrain cadence; fraud=days, image=years
Not real
Constant

Model decay: rolling accuracy + vintage curve + survival; informs retrain.

#monitoring

215 interview questions in this topic.