EasyDeepLearn
Supervised Learning · section 17 of 18

Pipelines & leakage

8 interview questions on pipelines & leakage, each answered in full. Free to read, no account needed.

Why should preprocessing live inside a scikit-learn Pipeline rather than being applied manually before?

easy
  • Pipelines guarantee that every preprocessing step is fit on the *training fold* only (during CV, hyperparameter search, and grid search) and applied to validation/test.
  • Manual preprocessing before splitting is the classic source of leakage: a StandardScaler fit on the full data has already seen the test set's variance.
  • Pipelines also encapsulate the whole model into one deployable object — pickle it, deploy it, and inputs get transformed identically at serving time.
  • Non-negotiable for reproducibility and safety.
#pipelines#data-qualityPermalink & quiz →

Your new model has 60% AUC and the baseline had 75%. How do you debug?

medium
  • Systematic checklist: (1) verify label correctness on validation — often the bug; (2) check for data leakage in the *baseline* (not the new model); (3) confirm train/val came from the same distribution — look for schema drift, temporal drift, missing categories; (4) inspect learning curves: is training loss even decreasing?
  • (5) rule out preprocessing issues (unscaled features, wrong encoding, target leakage after refactor); (6) sanity-check with a very simple model (logistic regression) on the same features and pipeline.
  • Only after this should you tune hyperparameters.
#interview-scenarios#data-qualityPermalink & quiz →

You detect concept drift in production. What do you do?

medium
  • (1) Quantify: is it data drift (features change) or concept drift (relationship  P(y∣x)  changes)(\mathrm{relationship}\;P(y \mid x)\;\mathrm{changes})?
  • Different fixes.
  • (2) Short-term: fall back to a safer model (simpler, better-calibrated) or raise your decision threshold conservatively.
  • (3) Medium-term: retrain on recent data with a rolling window.
  • (4) Long-term: instrument monitoring (population stability index, calibration on live labels) to catch it earlier next time and consider online / incremental learning.
  • Never silently retrain — always human-review a drifted model before it ships back to production.
#interview-scenarios#data-qualityPermalink & quiz →

How do you train and evaluate a model when labels arrive weeks after predictions?

hard
  • (1) Choose an evaluation delay matching the label horizon — 'test set is data from more than N weeks ago'.
  • (2) Never mix current features with future-only labels — check every feature for time-of-availability.
  • (3) Use only features available at prediction time (build a snapshot feature store or historical replay).
  • (4) Sample the delayed-label distribution: some labels may never arrive; account for censoring (survival analysis) or the label-noise it introduces.
  • (5) Monitor a proxy signal (click, conversion micro-events) in the meantime so you notice problems before the true labels land.
#interview-scenarios#data-qualityPermalink & quiz →

Your model has great offline metrics. What do you check before serving it in production?

medium
  • (1) Latency at target p99 under realistic load.
  • (2) Memory / cost at expected traffic.
  • (3) Fairness / calibration by subgroup — global metrics can mask per-segment failures.
  • (4) Compare with the current model on a shadow deployment before A/B testing — offline metrics almost never fully match live traffic.
  • (5) Rollback plan: canary + circuit breaker.
  • (6) Feature parity between training and serving pipelines — training-serving skew is the single most common production ML bug.
  • (7) Monitoring for input drift + prediction drift + business KPI in the same dashboard.
#interview-scenarios#pipelinesPermalink & quiz →

Your model scores AUC 0.92 offline but barely helps in production. What are the usual causes?

hard
  • Leakage is the first suspect: a feature that encodes the future, so it exists in training but not at inference.
  • Second, training / serving skew — the feature is computed differently by the training job and the serving path.
  • Third, a distribution shift between the training window and live traffic.
  • Fourth, the offline metric measures the wrong thing: AUC over all users while the product only acts on the top 1%, where precision is what matters.
  • Fifth, the intervention itself is weak — the model ranks well but the action taken on the prediction does not change behaviour.
#interview-scenarios#data-qualityPermalink & quiz →

How do you handle noisy labels in a supervised dataset?

hard
  • First quantify it: relabel a random sample of a few hundred rows yourself and measure the disagreement rate, which bounds the accuracy any model can reach.
  • Then reduce sensitivity: prefer robust losses such as Huber for regression, use label smoothing for classification, and avoid training to zero error since heavy overfitting memorizes the noise.
  • Confident learning approaches, such as cleanlab, rank rows whose predicted distribution strongly disagrees with the given label so a human can review the worst offenders.
  • Finally, keep a small hand-audited gold set for evaluation — you cannot measure progress against noisy labels.
#data-quality#interview-scenariosPermalink & quiz →

How do you decide how often to retrain a supervised model?

medium
  • Measure rather than guess.
  • Take the current model, evaluate it on successive time slices of held-out data, and plot how performance decays with the age of the training window.
  • The shape of that curve tells you the cadence: if the metric is flat for three months, monthly retraining is waste.
  • Weigh it against cost and risk, since every retrain is a chance to ship a regression, so you need automated validation gates.
  • Add event-driven triggers on top of the schedule: a drift alarm, a known upstream schema change, or a product launch that shifts the population.
#interview-scenarios#data-qualityPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz