How do you monitor model calibration in production?

hard

Answer

$Bin predictions by predicted probability.$
$For each bin, compute observed frequency of positive outcome (needs labeled data).$
$(\mathrm{bin}_{\mathrm{weight}}\; \times \; \mid \mathrm{avg}_{\mathrm{pred}}\; - \;\mathrm{observed}_{\mathrm{freq}} \mid )$
$Plot reliability diagram.$
$Alert if ECE degrades.$
$Miscalibration causes: distribution shift, class imbalance change, threshold drift.$
$Recalibrate via Platt scaling / isotonic without retraining.$
$Critical for probabilistic decisions (loan approval thresholds).$

Check yourself — multiple choice

Random
Bin by predicted prob, compare observed freq per bin; ECE = Σ $\mathrm{bin}_{\mathrm{weight}}$ × |pred - observed|; reliability diagram; alert on ECE degrade; recalibrate via Platt/isotonic without retraining
Just accuracy
Not real

Calibration: ECE + reliability diagram + recalibrate on drift.

#monitoring

215 interview questions in this topic.