Your model has great offline metrics. What do you check before serving it in production?
mediumAnswer
- (1) Latency at target p99 under realistic load.
- (2) Memory / cost at expected traffic.
- (3) Fairness / calibration by subgroup — global metrics can mask per-segment failures.
- (4) Compare with the current model on a shadow deployment before A/B testing — offline metrics almost never fully match live traffic.
- (5) Rollback plan: canary + circuit breaker.
- (6) Feature parity between training and serving pipelines — training-serving skew is the single most common production ML bug.
- (7) Monitoring for input drift + prediction drift + business KPI in the same dashboard.
Check yourself — multiple choice
- Just deploy and hope
- Latency, cost, per-segment calibration, shadow test, feature parity, rollback plan, monitoring
- Only check accuracy
- Offline metrics are always enough
Latency + cost + fairness + shadow test + feature parity + rollback + monitoring — always.
#interview-scenarios#pipelines
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- You must pick ONE metric for your model. How do you decide?
- Why should preprocessing live inside a scikit-learn Pipeline rather than being applied manually before?
- What does ColumnTransformer do and when do you need it?
- You did StandardScaler().fit_transform(X) then split into train/test. Why is this wrong?
- Name three situations where you should NOT reach for machine learning.
- Your new model has 60% AUC and the baseline had 75%. How do you debug?