Your new model has 60% AUC and the baseline had 75%. How do you debug?
mediumAnswer
- Systematic checklist: (1) verify label correctness on validation — often the bug; (2) check for data leakage in the *baseline* (not the new model); (3) confirm train/val came from the same distribution — look for schema drift, temporal drift, missing categories; (4) inspect learning curves: is training loss even decreasing?
- (5) rule out preprocessing issues (unscaled features, wrong encoding, target leakage after refactor); (6) sanity-check with a very simple model (logistic regression) on the same features and pipeline.
- Only after this should you tune hyperparameters.
Check yourself — multiple choice
- Immediately increase model capacity
- Label correctness → leakage → distribution shift → learning curves → preprocessing → simple-model sanity check
- Only tune hyperparameters
- Randomize the data
Debug bottom-up: labels → leakage → distribution → training dynamics → preprocessing → sanity model.
#interview-scenarios#data-quality
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- You detect concept drift in production. What do you do?
- How do you train and evaluate a model when labels arrive weeks after predictions?
- Your model scores AUC 0.92 offline but barely helps in production. What are the usual causes?
- How do you handle noisy labels in a supervised dataset?
- How do you decide how often to retrain a supervised model?
- How do you handle class imbalance in a dataset?