Your model scores AUC 0.92 offline but barely helps in production. What are the usual causes?
hardAnswer
- Leakage is the first suspect: a feature that encodes the future, so it exists in training but not at inference.
- Second, training / serving skew — the feature is computed differently by the training job and the serving path.
- Third, a distribution shift between the training window and live traffic.
- Fourth, the offline metric measures the wrong thing: AUC over all users while the product only acts on the top 1%, where precision is what matters.
- Fifth, the intervention itself is weak — the model ranks well but the action taken on the prediction does not change behaviour.
Check yourself — multiple choice
- The model is simply undertrained
- Leakage, training/serving skew, distribution shift, an offline metric that does not match the decision, or a weak downstream action
- AUC is always unreliable
- Production hardware is slower
Offline/online gaps come from leakage, skew, shift, metric mismatch or a weak intervention.
#interview-scenarios#data-quality
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Your new model has 60% AUC and the baseline had 75%. How do you debug?
- You detect concept drift in production. What do you do?
- How do you train and evaluate a model when labels arrive weeks after predictions?
- How do you handle noisy labels in a supervised dataset?
- How do you decide how often to retrain a supervised model?
- How do you handle class imbalance in a dataset?