Interview: your model's accuracy dropped 15% overnight — how do you diagnose?
hardAnswer
- (1) Rule out incidents first: pipeline break, feature-source outage, label logging bug.
- (2) Check input distributions: KS + PSI per feature vs baseline.
- (3) Check label distribution + prediction distribution shift.
- (4) Segment: is drop uniform or in specific segment?
- (5) Retrain on last 7 days as sanity check.
- (6) Post-mortem: if drift, decide short-term (rollback / hotfix threshold) vs long-term (retrain cadence + monitoring).
- (7) Add automated alert for the specific drift metric caught.
Check yourself — multiple choice
- Random
- Rule out infra bugs first + KS/PSI feature drift + label/pred drift + segment analysis + sanity retrain + rollback/hotfix + monitoring hardening
- Just retrain
- Not real
15% overnight drop: infra bug first + drift stats + segment + retrain + monitoring.
#interview#monitoring#applications
Practise Unsupervised Learning
214 interview questions in this topic.
Related questions
- Interview: how do you monitor drift in an embedding-based retrieval system?
- Interview: 'you're asked to segment 10M customers — how?'
- How do you monitor cluster drift over time?
- Interview: 'design an unsupervised fraud detection system'.
- Interview: cluster 5M e-commerce customers for a marketing campaign.
- Interview: production fraud detector — daily volume 10M, current FN too high.