72 interview questions on data quality & drift, each answered in full. Free to read, no account needed.
What is data drift and how do you detect it?
easy- Data drift is a change in the input feature distribution P(X) over time compared to training data.
- Detect it with statistical tests per feature (KS test for numeric, chi-square for categorical), Population Stability Index (PSI > 0.2 = significant drift), or distance measures (JS divergence, Wasserstein).
- Monitor drift in production dashboards and alert when thresholds are crossed.
- Drift does not always hurt performance — measure impact on output and label if available.
How is concept drift different from data drift?
medium- Concept drift is a change in the relationship P(y∣x) — the mapping from features to label evolves over time (customer behavior changes, fraud patterns evolve).
- Data drift changes P(X); label drift changes P(y).
- Concept drift is more dangerous because inputs may look normal but predictions become wrong.
- Detect it by monitoring model performance on labeled data or via delayed feedback and calibration checks.
What does 'data imbalance' mean and why is it a problem?
easy- Data imbalance means classes are unequally represented (e.g., 99% negatives, 1% positives).
- A naive model can hit 99% accuracy by always predicting the majority class, learning nothing about the minority.
- It's a problem when the minority class is the one you actually care about (fraud, rare disease).
- Handle with resampling (SMOTE, undersampling), class weights, threshold tuning, appropriate metrics (PR-AUC, F1, recall at fixed precision), and anomaly detection framing for extreme imbalance.
What are common sources of data leakage in an ML pipeline?
medium- (1) Fitting scalers/encoders on the full dataset before splitting.
- (2) Using future information (post-outcome features).
- (3) Grouping leakage: same user in both train and test.
- (4) Duplicated rows across splits.
- (5) Target-encoded features computed on the entire dataset.
- (6) Time-series data shuffled instead of split chronologically.
- Use sklearn Pipelines, grouped/time-aware splits, and thorough exploratory checks.
What should you monitor in a production ML model?
easy- (1) Prediction distribution over time.
- (2) Feature distributions and drift metrics.
- (3) Model performance vs delayed ground truth (accuracy, precision/recall).
- (4) Latency, throughput, error rates.
- (5) Data quality (missing values, out-of-range).
- (6) Business KPIs the model affects.
- Set alerts on drift and on performance drops.
- Log inputs, predictions, and outcomes for offline analysis and retraining.
How do you validate data quality in an ML pipeline?
medium- Set up automated checks with tools like Great Expectations, TFDV, or Deequ.
- Validate: schema (types, columns present), value ranges, allowed categories, null rates, uniqueness, referential integrity, and distributional stats (mean, std, quantiles) vs a reference.
- Fail loudly on violations, quarantine bad rows, and page on-call for critical breaches.
What is schema drift and how do you detect it?
medium- Schema drift = upstream data changes shape: column added / removed / renamed, dtype changes (int → string), enum values expanded.
- Silent failures because pipelines don't crash but predictions become garbage.
- Detect via schema validation on ingestion (Great Expectations, TFDV, Pandera): assert expected columns + types + allowed values.
- Fail pipeline loudly on mismatch; never fill with defaults silently.
- Modern practice: contract-based data pipelines where producers must declare schema changes.
What is label drift and why does it matter?
medium- Label drift = change in P(y) marginal distribution — class balance shifts over time.
- Example: fraud rate rises from 1% to 3%.
- Model's threshold + calibration may become miscalibrated even if P(y∣x) unchanged.
- Detect by tracking observed class ratios in ground truth over time.
- Handle via re-calibration (Platt / isotonic), threshold tuning, or class-weight re-adjustment.
- Different from data drift (P(X)) and concept drift (P(y∣x)).
How is Population Stability Index (PSI) computed and interpreted?
hard- Bin feature into k quantiles based on training distribution.
- Compute expected % (train) and actual % (current) per bin.
- PSI = Σ (actual - expected) × ln(actual/expected).
- Interpretation: <0.1 stable, 0.1-0.2 minor shift (investigate), >0.2 significant drift (retrain / alert).
- Widely used in banking / credit scoring.
- Symmetric version = JSD.
- Pitfall: small bins can inflate PSI — cap actual to 0.001 to avoid log-inf.
Why prefer Jensen-Shannon Divergence over KL for drift?
hard- KL(P∣∣Q) is asymmetric + unbounded + undefined when Q=0 for some support.
- JSD = ½ KL(P∣∣M) + ½ KL(Q∣∣M) where M = (P+Q)/2 is symmetric + bounded to [0, ln 2] (or [0,1] with base 2) + always defined.
- Interpretable as square of a distance (√JSD is a metric).
- Bounded scale makes thresholding across features straightforward.
- Standard choice for drift dashboards.
What is Maximum Mean Discrepancy (MMD) for drift detection?
hard- Kernel-based two-sample test.
- Embed both distributions in RKHS via kernel k.
- MMD2(P,Q) = ||μP−μQ||²_H = E[k(x, x')] - 2 E[k(x, y)] + E[k(y, y')].
- Zero iff P = Q.
- Handles high-dimensional data (unlike per-feature KS).
- RBF kernel with median-bandwidth heuristic works well.
- Permutation test for p-value.
- Foundation of modern high-dim drift (embeddings from LLMs / vision models).
When is Wasserstein distance appropriate for drift?
hard- Wasserstein-1 = 'earth mover's distance' — minimal work to move mass from P to Q.
- Advantages: (1) sensitive to shifts (unlike KL that focuses on mode overlap).
- (2) Works even when supports differ.
- (3) Metric (triangle inequality).
- (4) Meaningful units (same scale as feature).
- Use for continuous features with ordinal/spatial meaning (prices, temperatures).
- Drawback: expensive in high-D — use sliced Wasserstein or 1D-per-feature.
How do you monitor drift on unstructured data (text / images)?
hard- (1) Compute embeddings from pretrained model (CLIP for images, sentence-BERT for text).
- (2) Reduce to low-D via PCA / UMAP for viz.
- (3) Compute MMD / energy distance / Wasserstein between train and current embeddings.
- (4) Alert on threshold breach.
- (5) Sample outliers with high per-example score for human review.
- Complementary: monitor prediction confidence distribution + per-cluster prevalence.
Outliers vs drift — how do you distinguish?
medium- Outliers = individual points far from distribution; drift = the distribution itself shifts.
- Time-based test: (1) if score spikes then returns → outliers or anomalies.
- (2) if score persistently elevated → drift.
- Aggregate score over rolling window (24h).
- Individual outlier detection: Isolation Forest, LOF, autoencoder reconstruction error.
- Drift detection: aggregated PSI / MMD over batch.
- Both matter but require different responses.
How do you handle missing values in a production pipeline?
medium- (1) Categorize: MCAR (random), MAR (depends on observed), MNAR (depends on missing itself).
- (2) Simple: mean/median (numeric) or mode/'missing' category (categorical).
- (3) Model-based: KNN-imputation, iterative imputer (MICE), matrix completion.
- (4) Tree models handle missing natively (XGBoost, LightGBM).
- (5) Add binary 'ismissing' indicator feature — often predictive.
- (6) NEVER impute using future data — leaks.
- (7) Monitor missing rate per feature — spike = upstream issue.
How does imputation cause data leakage?
medium- Fitting imputer (mean / KNN) on train + test together sees test values → optimistic evaluation.
- Same for standardization / target encoding.
- Fix: fit imputer ONLY on training fold inside cross-validation; apply to test/validation via
.transform(). - Use
sklearn.pipeline.Pipeline or ColumnTransformer to enforce. - For time series: use train-window statistics only.
- Never use future observations to fill past.
How do you handle label noise in training data?
hard- (1) Confident Learning (cleanlab): identifies mislabeled examples via out-of-fold predictions.
- (2) Symmetric losses (MAE, generalized cross-entropy) more robust than CE.
- (3) Bootstrap / co-teaching: two networks vote.
- (4) Label smoothing (softens hard labels).
- (5) Active re-labeling: send uncertain / high-loss examples for human review.
- (6) Weighting by annotator reliability if multi-annotator.
- Empirically, models tolerate ~10-20% label noise; more requires cleaning.
How do you measure and improve annotation quality?
medium- (1) Inter-Annotator Agreement (IAA): Cohen's κ (2 annotators), Fleiss's κ (3+), Krippendorff's α.
- Below 0.6 = poor, above 0.8 = strong.
- (2) Gold-standard test tasks to score annotators.
- (3) Redundant annotation with adjudication for critical labels.
- (4) Clear guidelines + examples + edge cases.
- (5) Iterative refinement of guidelines based on disagreement analysis.
- (6) Regular calibration meetings.
- (7) MACE / dawid-Skene for probabilistic label aggregation.
What is active learning and when should you use it?
medium- Iteratively query most informative unlabeled examples for human labeling.
- Strategies: (1) Uncertainty sampling (max entropy / min margin / least confidence).
- (2) Query-by-committee (disagreement between ensemble).
- (3) Diversity sampling (core-set) to avoid redundant queries.
- (4) BALD (Bayesian mutual information).
- Use when labels are expensive (medical, legal, expert domains).
- Skip when labels are cheap (crowdsourcing) or class balance matters more than label count.
What is weak supervision (e.g., Snorkel)?
hard- Generate noisy labels programmatically from labeling functions (LFs): heuristics, keyword patterns, existing model outputs, distant supervision from KBs.
- Snorkel + Data Programming: LFs vote → generative model estimates LF accuracies + correlations → produces probabilistic labels → train final model on those.
- Trades label quality for volume — good when clean labels expensive but coverage matters (weak > few strong).
- Modern extension: prompting LLMs as an LF.
How does time-based leakage happen and how do you prevent it?
hard- (1) Using future timestamps in features (fitting on aggregate features computed with post-event data).
- (2) Random shuffle instead of time-based split.
- (3) Target encoding across time (using future target information).
- Fix: (1) time-based split (train ← past, test ← future).
- (2) Purged CV with embargo (gap between train/test to avoid autocorrelation).
- (3) Feature engineering only uses features known BEFORE prediction time (as-of-timestamps in feature store).
- (4) Backtesting with rolling / expanding windows.
What is purged cross-validation?
hard- López de Prado.
- Standard k-fold + (1) remove training samples whose labels depend on test period (purge).
- (2) Add embargo gap after test period (prevents leakage via serial correlation in labels/features).
- Essential when labels span multiple time bars (returns over next N days).
- Without purging, correlation between train and test through overlapping label windows inflates performance.
- Standard in quantitative finance.
SMOTE vs class weights vs threshold tuning — which to use?
medium- (1) Class weights: cheapest, no data manipulation.
- Try first — often sufficient for 10-100:1 imbalance.
- (2) Threshold tuning: adjust decision threshold on ROC curve to trade precision/recall — free (post-training).
- (3) SMOTE: interpolates synthetic minority points in feature space.
- Works when features are continuous + interpolation is meaningful.
- Risky in high-D + text.
- Modern variants (ADASYN, borderline SMOTE).
- (4) Undersampling majority: throws data away — combine with ensembles.
- Focal loss on deep models.
- Use threshold + weights first, SMOTE only if still insufficient.
Why version data (DVC / lakeFS / Delta Lake)?
medium- (1) Reproducibility: recreate any past experiment with exact input data.
- (2) Rollback: revert bad ingestion / labeling changes.
- (3) Blame: identify which upstream change broke a model.
- (4) Time-travel queries: 'what would model X have predicted on data as of Nov 1?'.
- Tools: DVC (git-like for files), lakeFS (git-like for object storage), Delta Lake / Iceberg (transactional table formats).
- Critical for regulated industries (audit trail).
How do you monitor fairness in production?
hard- (1) Track prediction / performance metrics per protected group (gender, race, age, geography).
- (2) Compute fairness metrics: demographic parity, equal opportunity, equalized odds, calibration by group.
- (3) Alert when disparity exceeds threshold.
- (4) Regular audits + reporting.
- (5) Log inputs + protected attributes for post-hoc analysis.
- (6) Store predictions with proxies (zip code, name) for indirect discrimination detection.
- Tools: Fairlearn, AIF360, What-If Tool.
How do you handle PII in ML pipelines?
hard- (1) Minimize collection: only fields needed.
- (2) Pseudonymization: hash identifiers.
- (3) Differential privacy: add calibrated noise (DP-SGD for training).
- (4) Federated learning: train on-device, share gradients (not data).
- (5) Access control + audit logs.
- (6) Encryption at rest + in transit.
- (7) Right-to-forget: pipelines that remove user data + retrain.
- (8) Model inversion / membership inference attacks: test with DP + regularization.
- Regulations: GDPR, CCPA, HIPAA (healthcare).
How does selection bias affect ML systems?
hard- Training data doesn't represent deployment population.
- Examples: (1) recommender only trained on items shown → can't learn about unshown items (exposure bias).
- (2) credit scoring: trained on approved applicants only → censored.
- (3) medical diagnosis: only diagnosed patients labeled.
- Fixes: (1) importance weighting toward deployment distribution.
- (2) inverse propensity scoring (IPS) for exposure bias.
- (3) exploration / random rollout to break selection.
- (4) counterfactual evaluation.
- (5) actively collect from underrepresented segments.
What is a model feedback loop and why is it dangerous?
hard- Model's predictions influence future training data → self-reinforcing.
- Example: (1) recommender pushes popular items → users click popular → more training signal for popular → runaway effect.
- (2) hiring algorithm reject certain profile → no future data for that profile.
- (3) fraud model detects one pattern → attackers shift, but model doesn't learn new pattern from its own denials.
- Detect via causal reasoning + exposure logs.
- Fix via exploration / random baseline traffic + counterfactual evaluation + off-policy correction.
What is a golden test set and how do you maintain it?
medium- Curated, high-quality, hand-labeled evaluation set that remains fixed across model versions — enables consistent comparison.
- Properties: (1) representative of production distribution.
- (2) covers edge cases + failure modes.
- (3) contains fairness / bias slices.
- (4) labeled by experts (not crowdsourced).
- (5) NEVER used for training (leakage risk).
- Maintenance: augment with new edge cases from production failures; periodic review + relabeling; freeze snapshot for each release.
What is a data contract?
medium- Formal agreement between data producer + consumer specifying: (1) schema (columns + types + nullability).
- (2) semantic meaning of each field.
- (3) update frequency + freshness SLA.
- (4) quality guarantees (uniqueness, range, referential integrity).
- (5) breaking-change process.
- Enforced via automated checks in CI on the producer side.
- Prevents 'silent schema drift'.
- Modern data engineering practice popularized by Andrew Jones.
- Tools: Data Contracts CLI, Iceberg schema evolution rules.
What is point-in-time correctness in feature stores?
hard- When constructing training data, for each event at time t, use only feature values known BEFORE t.
- Prevents leakage from future data into features.
- Requires: (1) features timestamped with validfrom+validto.
- (2) join uses 'as-of' semantics: max(featuretimestamp≤eventtimestamp).
- (3) Feature store must maintain event log, not just current snapshot.
- Without PIT, training features reflect future information that won't exist at serve time.
How do you guarantee online-offline feature consistency?
hard- (1) Single source of truth for feature definition (code + config).
- (2) Compute both paths from same definition: batch compute (offline) + streaming compute (online) share transformation logic.
- (3) Automated parity checks: sample production requests, recompute offline, compare.
- (4) Alert on divergence.
- (5) Feast / Tecton implement this via 'feature views'.
- (6) Ideal: same code runs in both contexts (Python function transpiled to SQL + streaming).
- Anti-pattern: hand-write both paths → drift.
What do you monitor about features (not just models)?
medium- (1) Freshness: is the feature updated on schedule?
- (2) Completeness: missing rate per feature.
- (3) Distribution drift: PSI / MMD vs baseline.
- (4) Cardinality: for categorical, new unseen values.
- (5) Correlation stability: pairwise correlations vs baseline.
- (6) Feature importance stability: SHAP over time.
- (7) Range violations: values outside expected.
- Store per-feature health score → dashboard.
- Predictive: feature health degrades before model does.
How do you structure a progressive rollout with automated guardrails?
hard- (1) Deploy to 1% traffic.
- (2) Monitor N minutes: error rate, latency p50/p99, business KPI, model-specific metrics (calibration).
- (3) If any guardrail exceeded → auto-rollback + page.
- (4) Otherwise auto-advance to next stage (5% → 25% → 50% → 100%).
- (5) Each stage has minimum bake time.
- Tools: Flagger, Argo Rollouts, LaunchDarkly.
- Metrics + thresholds pre-configured before rollout.
- No human intervention if all green.
Why do offline improvements often not translate online?
hard- (1) Distribution shift: offline eval on historical data doesn't reflect current traffic.
- (2) Feedback effects: recsys changing displayed items changes user behavior.
- (3) Metric mismatch: offline accuracy vs online business KPI.
- (4) Latency degradation: model too slow at production scale.
- (5) Serving-training skew: features computed differently.
- (6) Novelty effect: users react to change itself, not model quality.
- (7) Long-tail failures: rare edge cases dominate real users.
- Always validate offline gains with online A/B.
SLO / SLI / SLA for ML services.
medium- SLI (Service Level Indicator): metric measured (latency p99, error rate, accuracy).
- SLO (Objective): target for SLI (p99 < 100ms 99.9% of time).
- SLA (Agreement): contractual commitment to customer with financial penalties if breached.
- ML-specific SLIs: prediction latency, freshness of features, model accuracy vs holdout, drift score.
- Error budget = 1 - SLO: how much unreliability tolerated before halting risky deploys.
- Standard SRE framework applied to ML.
How do you avoid alert fatigue in ML monitoring?
medium- (1) Prioritize: only alert on things needing immediate action; log others to dashboard.
- (2) Escalation: warnings → email; errors → Slack; critical → page.
- (3) Deduplication: don't fire same alert twice in N minutes.
- (4) Correlate: parent alert suppresses child (drift + accuracy drop → single alert).
- (5) Runbook per alert: what to do.
- (6) Weekly review: alerts fired vs actioned; delete useless ones.
- (7) Sensitivity tuning: minimize FP.
- Rule: if alert can't be actioned, it shouldn't fire.
Three pillars of observability applied to ML.
medium- (1) Metrics: aggregated numeric signals over time (Prometheus).
- ML: prediction distribution, latency, drift score, feature freshness.
- (2) Logs: structured events with context (Loki, ELK).
- ML: prediction logs with input + output + version + latency.
- (3) Traces: causal chain of a single request across services (Jaeger, Tempo, Datadog APM).
- ML: feature fetch → preprocess → model call → postprocess.
- Modern add: profiles (continuous profiling).
- Enable debugging beyond dashboards.
How do you monitor feature / prediction freshness?
medium- Freshness = time between event and its reflection in served feature.
- Monitor: (1) end-to-end lag: max(currenttime−featureupdatedat) across features.
- (2) SLA per feature (real-time features: seconds; daily features: hours).
- (3) Alert when lag exceeds SLA.
- (4) Root causes: streaming pipeline lag, batch job failed, source system down.
- Tools: Kafka lag metrics (consumer group), Airflow SLA, feature-store metadata.
- Critical for time-sensitive predictions (fraud, recsys).
How does shadow evaluation work?
medium- New model runs on production requests in parallel with current model.
- Predictions logged, not served.
- Compute: (1) agreement rate with production model.
- (2) predicted-metric distribution vs production.
- (3) latency + memory under real load.
- (4) if labels available (delayed), online accuracy.
- Purpose: safety check before A/B — catch latency issues, prediction anomalies, correlations with production.
- No user impact.
- Typical duration: 1-7 days.
- Standard pre-A/B step.
What is Canary Analysis (Kayenta / Flagger)?
hard- Automated statistical comparison between canary + baseline metrics during progressive rollout.
- Netflix Kayenta: (1) fetch metrics for canary + baseline over rollout window.
- (2) run Mann-Whitney U or similar test per metric.
- (3) assign score per metric + weighted total.
- (4) promote if above threshold; rollback if below.
- Removes human bias.
- Modern: Argo Rollouts + Prometheus queries + web hooks.
- Enables auto-rollout with confidence.
How do you monitor prediction uncertainty in production?
hard- (1) Log softmax entropy / max prob per prediction.
- (2) MC dropout / ensemble variance for Bayesian models.
- (3) Aggregate: histogram of uncertainty over time.
- (4) Alert if mean uncertainty spikes → OOD input distribution.
- (5) Route high-uncertainty predictions to human review or fallback.
- (6) Selective prediction: refuse when uncertainty > threshold, defer to human.
- Critical for medical / legal / financial ML.
How do you monitor model calibration in production?
hard- Bin predictions by predicted probability.
- For each bin, compute observed frequency of positive outcome (needs labeled data).
- Expected Calibration Error (ECE) = Σ (binweight×∣avgpred−observedfreq∣).
- Plot reliability diagram.
- Alert if ECE degrades.
- Miscalibration causes: distribution shift, class imbalance change, threshold drift.
- Recalibrate via Platt scaling / isotonic without retraining.
- Critical for probabilistic decisions (loan approval thresholds).
Why monitor per-slice performance?
medium- Overall accuracy can hide subgroup regressions.
- Monitor per (country, device, user tier, time of day, product category).
- Identify: (1) systematic bias (accuracy drops in minority group).
- (2) local drift (feature shift in one region).
- (3) failure modes (specific product types).
- Alert on per-slice metric drop > X% even when overall stable.
- Complementary: fairness metrics per protected group.
- Tools: Fiddler AI, Arize, WhyLabs.
Popular ML observability tools?
easy- (1) Evidently AI: open-source drift + performance reports.
- (2) WhyLabs / WhyLogs: streaming profile + monitoring.
- (3) Fiddler AI: enterprise explainability + monitoring.
- (4) Arize AI: production monitoring + fairness.
- (5) Aporia: monitoring + explainability.
- (6) Grafana + Prometheus with custom exporters.
- (7) Datadog ML monitoring.
- (8) Neptune / MLflow limited monitoring.
- Choose by: SaaS vs self-hosted + LLM support + integrations + budget.
How do you detect anomalous predictions in production?
medium- (1) Statistical: z-score of prediction / feature vs recent history.
- (2) Isolation Forest / LOF on request features.
- (3) Autoencoder reconstruction error on inputs.
- (4) Prediction confidence: sudden increase in low-confidence predictions.
- (5) Feature range violations (values outside training distribution).
- (6) Model-specific: attention weights unusual for LLMs. Aggregate to hourly / minute counts + alert on spike.
- Route anomalies to sample store for review.
How do you set thresholds for drift alerts?
hard- (1) Compute baseline: 1-2 weeks of production data during known-good period.
- (2) Set threshold: mean + 3σ, or specific PSI > 0.2, or KS p < 0.01.
- (3) Test on historical incidents: does threshold fire correctly?
- (4) Tune sensitivity vs FP rate.
- (5) Different thresholds per feature (some naturally noisier).
- (6) Auto-adjust seasonal: recompute baseline weekly.
- (7) Multi-scale: hourly + daily + weekly windows.
- Fixed thresholds get stale.
How do you monitor performance when labels are delayed?
hard- Ground truth arrives days / weeks / months later (loan default, customer churn).
- Strategies: (1) proxy metrics available immediately (clicks for CTR, purchase within 1h).
- (2) short-window versions of long-term label (7-day approx of 90-day churn).
- (3) leading indicators from user behavior.
- (4) partial evaluation on subset with fast feedback.
- (5) rolling window backfill: compute metric as labels arrive.
- (6) Alert on distribution changes even without labels.
- Match business KPI once available.
How do you monitor drift when data has natural seasonality?
hard- (1) Baseline per time period (hour-of-day, day-of-week, month) not one static.
- (2) Detrend via time-series decomposition (STL, Prophet) before comparing.
- (3) Compare same-period-last-year rather than yesterday.
- (4) Rolling window baselines auto-adapt.
- (5) Distinguish 'expected variation' from 'true drift' via multi-window ensemble.
- Anti-pattern: alert every Monday morning because Monday differs from Sunday.
What is model decay and how do you measure it?
medium- Gradual degradation of model performance over time due to drift + concept change + world evolving.
- Measure: (1) rolling accuracy on freshly-labeled batches.
- (2) accuracy vs training vintage.
- (3) compare snapshot models: retrain monthly, evaluate on latest data — degradation curve.
- (4) survival analysis on 'time until performance drops X%'.
- Decay rate informs retraining cadence.
- Some models decay in days (fraud), others in years (image classifiers on stable domains).
How do you respond to a production ML incident?
medium- (1) Alert fires → on-call paged.
- (2) Assess severity: user impact, revenue, safety.
- (3) Mitigate first: rollback / disable model / fallback to simpler model.
- (4) Root cause after: which change (code / data / traffic) triggered?
- (5) Communicate to stakeholders.
- (6) Post-mortem: timeline, contributing factors, action items.
- (7) Prevent: automated test, monitor, or process change.
- Rule: mitigate before diagnosing; don't debug live prod.
- Blameless post-mortem.
How do you monitor ML infrastructure cost?
medium- (1) Tag resources by (team, project, model, environment).
- (2) Cost per model per day: infra cost + prediction volume → /prediction.(3)Alertsonunexpectedcostspikes(>2σ).(4)Budgetalertsperteam.(5)FinOpsdashboard:idleGPU,over−provisionedpods,expensive−but−unusedfeatures.(6)Right−sizingrecommendations:instancetypes+autoscalebounds.(7)Costperexperiment(trainingrun).Tools:AWSCostExplorer,GCPBilling,Kubecost.Businessimpactper spent = key metric.
How do you handle biased ground truth collection?
hard- Feedback often depends on model output.
- Recsys: only see clicks on shown items — no signal for unshown.
- Fixes: (1) exploration: show random items occasionally for unbiased data.
- (2) inverse propensity scoring (IPS): weight by 1/P(shown).
- (3) counterfactual estimator.
- (4) small holdout randomly sampled.
- (5) explicit ratings vs implicit.
- Interview red flag: candidate optimizes offline metrics on biased data + surprised when online is different.
How do you explain individual predictions in production?
medium- (1) SHAP: computes Shapley values per feature contribution.
- TreeSHAP fast for trees.
- (2) LIME: local linear approximation.
- (3) Integrated gradients / Grad-CAM for deep models.
- (4) Attention weights for transformers (interpretation with caution).
- (5) Feature importance dashboards.
- (6) Counterfactual explanations: 'if X had been Y, prediction would be Z'.
- (7) Store per-prediction explanations for regulatory / customer service.
- Tools: SHAP, Captum, InterpretML.
How do you compute SHAP at production scale?
hard- SHAP is expensive (2Nfeaturesubsets).
- Optimizations: (1) TreeSHAP: exact+O(TLD2) for tree ensembles.
- (2) DeepSHAP: gradient-based for NNs.
- (3) KernelSHAP with sampling for model-agnostic.
- (4) Precompute + cache global feature importance.
- (5) Compute per-prediction only when requested (customer-facing explanation) or async.
- (6) Sample requests + summary dashboard.
- Alternative: SAGE for global, faster than SHAP.
The four golden signals — Google SRE.
easy- (1) Latency: time to serve request (p50, p99).
- (2) Traffic: requests per second.
- (3) Errors: rate of failed requests.
- (4) Saturation: how full resource is (CPU / memory / disk / GPU).
- Original SRE book.
- For ML add: (5) prediction quality / accuracy.
- (6) drift score.
- Together = comprehensive service health.
- Standard entry-point for observability dashboards.
- Every service should have all four.
RED vs USE vs Golden Signals — which methodology?
medium- RED (Requests / Errors / Duration): request-driven services (API, ML inference).
- USE (Utilization / Saturation / Errors): resource-focused (CPU, disk, GPU).
- Golden Signals (Google SRE): Latency / Traffic / Errors / Saturation — combines both perspectives.
- Best practice: RED for services + USE for infrastructure + Golden Signals as top-level.
- All three complement; not either/or.
Why use latency heatmaps instead of averages?
medium- Average / median hides bimodal or long-tail distributions.
- Heatmap: histogram over time (Y = latency bucket, X = time).
- Reveals: (1) tail spikes not visible in p50.
- (2) bimodality (fast path + slow path).
- (3) periodic patterns (batch job every 5 min).
- (4) shift over time.
- Standard in Prometheus / Grafana.
- Complementary metrics: p99, p99.9.
- Rule: latency has weird distribution — always histogram.
Common bad-alert patterns to avoid.
medium- (1) Alert on threshold that fires daily (noise).
- (2) Alert without runbook (no action).
- (3) Alert on symptoms, not causes ('page fires slower' rather than 'db p99 up').
- (4) Alerting on averages (hides tails).
- (5) Static thresholds ignoring seasonality.
- (6) No de-duplication (spam).
- (7) Waking humans for auto-recoverable.
- (8) Missing correlation (10 alerts for one root cause).
- (9) Alert on things with no fix.
- Rule: every alert = 'someone must do something now'.
How do you monitor GPU utilization?
medium- (1)
nvidia-smi real-time. - (2)
dcgm-exporter for Prometheus (per-GPU metrics: SM utilization, memory, temp, power). - (3) PyTorch Profiler / TensorBoard for training-time analysis.
- (4) NCU / Nsight for kernel-level.
- Key metric: SM occupancy — not just GPU busy %.
- Low occupancy = kernels not filling GPU (small batch, PCIe bottleneck, CPU bound).
- Interview: '100% utilization' can hide inefficient kernels; profile to find idle SMs.
How do you evaluate LLM outputs in production?
hard- (1) LLM-as-judge: strong model rates outputs on rubric (accuracy, helpfulness, safety) — cheap + fast, but bias toward similar-style output.
- (2) Golden set: hand-labeled Q&A / rubric — expensive but reliable.
- (3) Reference-based metrics (BLEU / ROUGE / BERTScore) for translation / summarization.
- (4) Automatic: exact match / regex / JSON validity.
- (5) Human eval periodic sample.
- (6) Downstream signals: click / thumbs up / task completion.
- (7) Guardrails: refusal, PII, harmful content detection.
- Combine multiple.
What are the biases of LLM-as-judge?
hard- (1) Position bias: prefers first / second response systematically → randomize order + swap.
- (2) Verbosity bias: prefers longer answers → normalize length.
- (3) Self-preference: judge favors output style similar to own model → use different judge model.
- (4) Familiarity bias: agrees with popular / common answers.
- (5) Rubric interpretation drift over runs.
- (6) Correlation with actual quality < 0.9 typically.
- Mitigate: multi-judge ensemble, human calibration, position swap, rubric with examples.
- Track judge-vs-human on golden set.
How do you monitor a RAG pipeline?
hard- (1) Retrieval metrics: recall@k, precision@k, MRR — needs labeled query-doc pairs.
- (2) Query embedding drift: distribution shift over time.
- (3) Index freshness: age of most recent doc, ingestion lag.
- (4) Retrieval quality proxy: does model use retrieved docs (grounding)?
- (5) End-to-end: answer quality via LLM-judge / golden set.
- (6) Cost: tokens per query (retrieval + generation).
- (7) Latency breakdown: embed + search + generate.
- (8) User: click through, thumbs.
- Tools: LangSmith, Langfuse, Arize LLM.
How do you observe LLM agents?
hard- Multi-step, tool-using agents: (1) Trace each step: user request → LLM call → tool call → response → LLM call → ... final answer.
- (2) Log tokens per step, latency per step, tool errors.
- (3) Full trace tree per session (LangSmith, Langfuse, Weave).
- (4) Success rate per intent (task completion).
- (5) Cost per session (accumulated tokens).
- (6) Debug trace when failure.
- (7) Golden set of test scenarios.
- Agents often fail silently — observability is critical, more than for single-shot LLM.
LangSmith / Langfuse — what do they provide?
medium- LangChain's LangSmith + open-source Langfuse: LLM-specific observability + eval.
- (1) Trace every LLM call, chain, agent step.
- (2) Prompt registry with versioning.
- (3) Eval datasets + automatic scoring.
- (4) User feedback annotation.
- (5) A/B experiment tracking.
- (6) Cost + latency dashboards.
- (7) Debug UI to inspect full trace.
- Alternatives: W&B Weave, Helicone, Portkey, PromptLayer, Arize Phoenix.
- Standard for LLM app development.
You inherit a model in production with no documentation. What do you check in your first week?
medium- Establish what it does before touching anything.
- Find the inference path and confirm which artefact is actually being served, since the deployed version is often not the one in the repository.
- Check whether the training data can be reconstructed, because a model you cannot retrain is a liability regardless of its accuracy.
- Compare the current input distribution against whatever the model was trained on, which usually reveals drift nobody was watching.
- Verify that predictions are logged with their inputs and a model version, since without that you cannot debug anything.
- Find out how outcomes are eventually observed, as that determines whether you can measure real performance at all.
- Only then look at the model itself.
Accuracy has probably dropped but labels arrive 60 days late. What can you monitor now?
hard- Everything upstream of the label.
- Input drift per feature, using a population stability index or a Kolmogorov-Smirnov test against a fixed training reference, catches the changes most likely to matter.
- Prediction drift is often the strongest single signal, because a shifted score distribution means either inputs changed or the model is behaving differently.
- Add proxy outcomes available sooner than the real label, such as a click when the target is a 60-day conversion, and monitor their relationship to predictions.
- Watch data quality separately, since null rates and cardinality changes precede most silent failures.
- Then check for concept drift indirectly by comparing calibration on whatever labels do trickle in early, accepting that they are a biased sample.
Your drift monitoring fires 40 alerts a day and everyone ignores it. How do you fix it?
medium- Alert on impact, not on statistics.
- A statistical test on a large sample detects differences too small to matter, so add a minimum effect size and require the drift to persist over several windows before firing.
- Rank features by their contribution to the model, because drift in a feature with negligible importance is not an incident, and drop the unimportant ones from paging entirely.
- Consolidate: one alert per model per day summarizing what moved beats forty independent alerts.
- Then tie the threshold to a decision, so every page has an action attached; if nobody would do anything differently, it belongs on a dashboard, not in a pager.
- Finally, review fired alerts monthly and delete the ones that never led to action.
How do you prove there is no training-serving skew?
hard- Compare the actual feature vectors, not the code that computes them.
- Log the features used at inference time, then take the same entities and recompute their features through the training pipeline for the same timestamp, and diff the values.
- Any mismatch is skew, and the common causes are a transformation implemented twice in two languages, a default value that differs, timezone handling, and a training pipeline that saw data updated after the event.
- Do this continuously on a sample rather than once, since skew appears whenever either pipeline changes.
- The structural fix is a single feature computation path shared by both, which is the main reason feature stores exist, and shadow scoring the training pipeline against production is the strongest ongoing check.
What is point-in-time correctness, and how does violating it look in practice?
hard- It means every feature value used for a training example reflects only what was knowable at that example's timestamp.
- Violating it produces leakage that is invisible offline: the model looks excellent in validation and disappoints in production, because at serving time the future information it relied on does not exist yet.
- The classic mechanisms are joining against a table that stores only the current value of an attribute, aggregating over a window that extends past the prediction time, and backfilling a column after an outcome was known.
- Detection is mostly structural, by reviewing every join for whether it filters on event time, and the pragmatic test is a suspiciously strong feature: an importance ranking dominated by one column usually means leakage rather than luck.
Scheduled retraining or triggered retraining?
medium- Scheduled is the sane default because it is predictable, testable, and forces the retraining path to stay working, which is the failure nobody notices until an emergency.
- Its weakness is that it retrains when nothing changed, wasting compute, and does not react between runs.
- Triggered retraining on a drift or performance signal reacts faster but introduces a nasty risk: an automated retrain on corrupted data deploys the corruption, so a trigger must be paired with data validation and an evaluation gate that can refuse to promote.
- In practice run both, a regular cadence plus a trigger for material degradation, and always gate promotion on an offline evaluation against a fixed benchmark rather than promoting whatever the pipeline produced.
What do you monitor for an LLM feature that you would not monitor for a classifier?
hard- Output-side quality signals, because there is no single correct label to compare against.
- Track refusal and empty-response rates, which spike when a prompt template or provider changes.
- Track groundedness for retrieval features, meaning the fraction of claims supported by the retrieved context, usually with an automated judge validated against human labels.
- Track token counts and cost per request, since these drift with user behaviour and directly hit the budget.
- Track latency to first token separately from total latency because they have different causes.
- Log the prompt version, model version and retrieved document identifiers with every request, since without them a regression is undebuggable.
- And sample real conversations for human review on a fixed cadence, which remains the highest-signal monitoring available.