What should you monitor in a production ML model?
easyAnswer
- (1) Prediction distribution over time.
- (2) Feature distributions and drift metrics.
- (3) Model performance vs delayed ground truth (accuracy, precision/recall).
- (4) Latency, throughput, error rates.
- (5) Data quality (missing values, out-of-range).
- (6) Business KPIs the model affects.
- Set alerts on drift and on performance drops.
- Log inputs, predictions, and outcomes for offline analysis and retraining.
Check yourself — multiple choice
- Only latency and errors matter
- You should monitor drift, performance, latency, and data quality together
- Delayed ground truth is useless
- Business KPIs are not part of ML monitoring
Real ML monitoring is multi-layer: performance + drift + system + business.
#monitoring
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- What is data drift and how do you detect it?
- How is concept drift different from data drift?
- What is schema drift and how do you detect it?
- What is label drift and why does it matter?
- How is Population Stability Index (PSI) computed and interpreted?
- Why prefer Jensen-Shannon Divergence over KL for drift?