Your drift monitoring fires 40 alerts a day and everyone ignores it. How do you fix it?
mediumAnswer
- Alert on impact, not on statistics.
- A statistical test on a large sample detects differences too small to matter, so add a minimum effect size and require the drift to persist over several windows before firing.
- Rank features by their contribution to the model, because drift in a feature with negligible importance is not an incident, and drop the unimportant ones from paging entirely.
- Consolidate: one alert per model per day summarizing what moved beats forty independent alerts.
- Then tie the threshold to a decision, so every page has an action attached; if nobody would do anything differently, it belongs on a dashboard, not in a pager.
- Finally, review fired alerts monthly and delete the ones that never led to action.
Check yourself — multiple choice
- Raise all thresholds to 0.99
- Require a minimum effect size and persistence, weight features by importance, consolidate into one digest per model, attach an action to every page, and prune alerts that never lead to action
- Remove monitoring
- Alert on every feature equally
Effect size, persistence, importance weighting and an attached action turn noise into signal.
#monitoring#observability
Practise MLOps & Data Quality
215 interview questions in this topic.