35 interview questions on deployment & experimentation, each answered in full. Free to read, no account needed.
What is training-serving skew?
medium- Training-serving skew is a mismatch between how features are computed at training and at serving time — different preprocessing code, different data sources, different schemas, different aggregations.
- Result: production performance is worse than offline eval.
- Prevent with a shared feature pipeline (feature store), unit tests that verify equivalent outputs, schema validation, and shadow-mode deployments before going live.
What is a shadow deployment and how does it differ from A/B testing?
medium- In shadow deployment, the new model runs alongside the current one on real production traffic, but its predictions are logged and not served to users.
- You compare its outputs to the production model without user impact.
- A/B testing splits real users between models and compares business metrics.
- Use shadow first to verify safety and correlations, then A/B to measure real impact.
What is a canary release for ML models?
easy- Canary release routes a small percentage of traffic (e.g., 1-5%) to the new model, monitors metrics and errors closely, and gradually increases traffic if healthy.
- If a regression is detected, you roll back quickly.
- Combines with automated guardrails on latency, error rate, and business metrics.
- Standard modern deployment strategy for both ML models and general services.
Batch vs online inference — how do you choose?
easy- Batch inference precomputes predictions on a schedule (nightly scoring) — cheap, high-throughput, tolerates seconds/minutes of latency.
- Online inference serves predictions in real time via HTTP/gRPC — needed for user-facing systems (recommendations, fraud checks).
- Choose based on latency budget and freshness need.
- Hybrid: online model with cached batch features from a feature store.
How do you A/B test a model rigorously?
hard- Randomly assign users to control (old model) and treatment (new model).
- Define the primary business metric before starting.
- Compute sample size needed to detect the expected effect with acceptable power.
- Run for a full business cycle (7+ days).
- Analyze with proper significance tests, account for multiple comparisons, and check guardrail metrics (latency, revenue, complaints).
- Stop early only via pre-specified sequential rules.
MLflow model registry — what does it provide?
medium- Centralized store for model artifacts + metadata.
- Features: (1) staging → production lifecycle.
- (2) version history.
- (3) approval workflow.
- (4) tags (champion / challenger).
- (5) model signature (inputs + outputs schema).
- (6) linked run + git commit + data version.
- Serves as source of truth for 'what's deployed'.
- Alternatives: BentoML, Vertex AI Model Registry, SageMaker Model Registry, Weights & Biases Artifacts.
What is a model signature and why does it matter?
medium- Explicit schema of model inputs + outputs: field names, types, shapes.
- Enables: (1) validation at serve time (reject invalid inputs).
- (2) auto-generated API + docs.
- (3) preventing training-serving skew (compare training schema to serve schema).
- (4) type-safe deployment.
- Standards: MLflow ModelSignature, ONNX metadata, OpenAI schema.
- Include example inputs + outputs.
- Version signature with model.
What is a blue-green deployment for ML models?
medium- Maintain two identical production environments: 'blue' (current) + 'green' (new).
- Deploy new model to green, run smoke + integration tests, then flip load balancer to green.
- Old blue kept warm for rapid rollback.
- Advantages: instant switchover + safe rollback + zero-downtime.
- Disadvantages: 2x infrastructure cost during switchover.
- Common for critical services; less common for ML where canary is preferred.
Canary vs blue-green — when to use each?
medium- Canary: gradual traffic shift (1% → 10% → 50% → 100%); great for ML where model quality is uncertain + wanting real-user metrics before full rollout.
- Blue-green: instant flip; great when new model was thoroughly tested offline + rollback speed matters more than gradual verification.
- ML preference: canary (or shadow + canary) because offline metrics don't capture everything (business KPI, latency under load).
- Blue-green for infrastructure-only changes.
What is champion-challenger pattern?
medium- Current production model = 'champion'.
- New candidate = 'challenger'.
- Both run in shadow or A/B; challenger is promoted to champion if metrics beat by significant margin over sufficient time.
- Multiple challengers can compete simultaneously (banking + credit models).
- Modern extension: multi-armed bandit auto-routes traffic to best-performing model with exploration.
- Standard pattern in credit / fraud / recsys.
A/B test vs multi-armed bandit — which to use?
hard- A/B: fixed allocation, wait for significance, use when you need clean causal comparison + all variants matter equally.
- Bandit (Thompson sampling / UCB): dynamically allocate more to winning variants — minimizes regret.
- Use when (1) many variants (10+), (2) speed matters over precision, (3) revenue impact during test matters.
- Downside: unbalanced sample → weaker inference.
- Modern: contextual bandit for personalized allocation.
- Netflix, Google use both.
What is interleaving in recsys A/B testing?
hard- Alternative to standard A/B for ranking systems.
- Instead of splitting users, merge rankings from both models into one presented list (team-draft or probabilistic interleaving).
- User clicks reveal preference.
- Advantages: (1) 10-100x more sample-efficient than user split.
- (2) Same user compares both.
- (3) Fast decisions.
- Used by Netflix, Bing, Yahoo.
- Downside: only measures ranking preference, not business KPI at page level.
What are guardrail metrics in an A/B test?
medium- Metrics you don't want to regress even if primary metric improves.
- Examples: (1) latency p99 (should not degrade).
- (2) error rate.
- (3) revenue (never negative).
- (4) safety violations (harmful content).
- (5) fairness metrics per group.
- (6) user complaints / bounce rate.
- Set two-sided bounds: primary must be significantly UP, guardrails must NOT be significantly DOWN.
- Pre-registered before test.
- Non-inferiority testing framework.
What is a sample ratio mismatch (SRM) alert?
hard- Actual traffic split (e.g., 51/49) differs significantly from designed (50/50).
- Chi-square test detects.
- Root causes: (1) buggy randomization / hashing.
- (2) selection bias (bot filtering asymmetric).
- (3) redirect skew (mobile vs desktop).
- (4) load imbalance (one variant fails silently, users bounce back).
- SRM invalidates the whole test — never analyze without fixing first.
- Standard automated check in any experimentation platform.
What is CUPED and why use it?
hard- Controlled Using Pre-Experiment Data.
- Adjust post-experiment metric Y by subtracting β × (X - E[X]) where X is pre-experiment metric for same user, β = Cov(X,Y)/Var(X).
- Reduces variance in metric → smaller sample size for same statistical power (often 30-50% reduction).
- Standard at Microsoft / Netflix / Booking.
- Prerequisite: pre-experiment data available per user.
- Extended: ML-CUPED uses ML model to predict Y from many pre-features.
How does stratified randomization help experiments?
hard- Randomize within strata (segments) — country, device, user tier.
- Guarantees balance on covariates.
- Reduces variance similar to CUPED.
- Analyzed via stratified estimator or regression with strata fixed effects.
- Especially useful for small experiments where random imbalance dominates.
- Netflix uses per-content-country stratification.
- Downside: more complex bookkeeping; too many strata → sparse cells.
Long-term holdout — what and why?
hard- Reserve 1-10% of users as permanent control who never receive new model / experiment.
- Measure long-term effects of everything shipped combined.
- Detect novelty / primacy effects, learned tolerances, and cumulative regressions that individual A/B tests miss.
- Standard at Google / Facebook / Netflix.
- Rotate holdout every 6-12 months.
- Small ongoing revenue cost, large diagnostic benefit.
How do you meet latency budgets in production ML?
hard- Budget = p50 or p99 target (e.g., 50ms).
- Techniques: (1) Model compression: quantization (int8/int4), pruning, distillation.
- (2) ONNX / TensorRT / Triton for optimized runtime.
- (3) Batching: dynamic batching increases throughput at slight latency cost.
- (4) Caching: memoize predictions for repeated inputs.
- (5) Two-stage: fast filter + slow rerank.
- (6) Async / precompute in shadow.
- (7) Hardware: GPU / TPU for large, CPU for small.
- Profile end-to-end; often preprocessing / network dominates, not model.
How do you control ML serving cost?
medium- (1) Right-size infrastructure: autoscale on utilization.
- (2) Batch requests where latency allows.
- (3) Distill to smaller model.
- (4) Cache popular predictions.
- (5) Two-stage: cheap filter + expensive rerank only on top-K.
- (6) Spot / preemptible instances for batch.
- (7) Compress model (quant / prune) to fit on cheaper HW.
- (8) Move batch scoring off critical path.
- (9) Serverless (Lambda / Cloud Run) for spiky traffic.
- (10) Multi-tenant serving: co-locate models on same GPU.
- Monitor $/prediction.
When is batch inference dramatically cheaper than online?
medium- Batch: (1) high GPU utilization via full batches (100-1000).
- (2) commodity hardware fine.
- (3) preemptible / spot instances OK.
- (4) no idle capacity.
- Online: (1) low batch (often 1) → low GPU utilization.
- (2) always-warm capacity for spikes.
- (3) latency-optimized (more expensive) hardware.
- Ratio: batch can be 10-100x cheaper per prediction.
- Use batch when predictions can be precomputed (nightly product recs, credit scores).
- Push to online store or DB.
How do you version model artifacts in production?
medium- (1) Semantic versioning (v1.2.3): major = incompatible schema, minor = new feature, patch = bug fix.
- (2) Store immutable artifacts in registry (S3 + metadata DB).
- (3) Tag with (git-commit, data-version, training-run-id).
- (4) Deployment references version, not 'latest'.
- (5) Multiple versions can serve simultaneously (traffic split).
- (6) Retention policy: keep last N + all promoted.
- (7) Include model signature (input / output schema) for compatibility checks.
What's a good rollback strategy?
medium- (1) Previous model version always available in registry.
- (2) Deployment supports instant revert (K8s rolling update revert, feature flag toggle).
- (3) Automated triggers: on-guardrail-breach → auto-rollback.
- (4) Manual: single-command / one-click.
- (5) Monitor rollback executed correctly (metric returns to baseline).
- (6) Post-mortem after rollback: what broke + how to prevent.
- Rule: rollback should be seconds to minutes, not hours.
- Practice quarterly.
How do you handle model errors in production?
medium- (1) Timeout: enforce max inference time; fallback if exceeded.
- (2) OOM protection: reject oversized inputs.
- (3) Input validation: schema + range checks.
- (4) Fallback model: simpler / cheaper model for edge cases or capacity issues.
- (5) Cached default: last-known-good prediction.
- (6) Log all errors with input for post-hoc analysis.
- (7) Circuit breaker: temporary disable if error rate spikes.
- (8) Graceful degradation over hard failure.
- Product should function even when model doesn't.
What is an A/A test and why run it?
medium- Both variants receive the SAME model.
- Expected result: no significant difference (~5% false-positive rate).
- Uses: (1) validate experimentation platform (SRM, randomization correctness).
- (2) baseline noise level for the metric.
- (3) sanity check before running A/B (ensure no accidental config difference).
- Run periodically.
- Standard practice at Airbnb, Booking, Microsoft.
- Failure = broken platform, don't trust any results until fixed.
How do network effects complicate A/B tests?
hard- User-level randomization assumes independence between users.
- Broken by: (1) social networks: friend sees new feed, tells friend in control.
- (2) marketplace: prices in one variant affect supply / demand for other.
- (3) messaging: control receives messages influenced by treatment.
- Fixes: (1) cluster randomization (whole regions / networks).
- (2) switchback experiments (whole population toggles by hour / day).
- (3) time-based split.
- Standard problem at Uber (drivers vs riders), Facebook (social graph).
Switchback experiments — when to use?
hard- Assign entire population to A or B alternately over time windows (e.g., every hour).
- Analyze paired periods.
- Handles: (1) network effects (whole market same variant).
- (2) marketplace balance.
- Downside: (1) confounded with time-of-day / weekday.
- (2) carryover: treatment period effects bleed into next control period (mitigation: exclude first N minutes).
- Uber uses for driver-side experiments.
- Careful analysis with mixed effects for time.
Difference-in-differences for ML experiments — when?
hard- When randomization impossible (e.g., can't ethically deny users, city-wide rollout).
- Compare treatment vs control unit change over pre-post period: DiD=(Ytreatpost−Ytreatpre)−(Yctrlpost−Yctrlpre).
- Assumes parallel trends absent treatment.
- Use when: (1) staggered rollouts across geographies.
- (2) policy changes affecting group.
- (3) natural experiments.
- Modern: synthetic control constructs weighted combination of control units.
How do you compute sample size for A/B?
medium- n ≈ 16 × σ2/(MDE)2 per arm for two-sided test at α=0.05 and 80% power.
- Inputs: (1) baseline metric variance σ2.
- (2) minimum detectable effect MDE.
- (3) significance α.
- (4) power (1 - β).
- Typical: 5% relative lift on a 10% baseline metric with p=0.05, 80% power needs 10K-100K users.
- Reduce sample via CUPED / stratification.
- Longer runs help capture weekly patterns but don't reduce sample requirement below noise.
How do you serve multiple models efficiently on shared infrastructure?
hard- (1) Multi-model server (TF Serving / Triton): load N models per container; route by request.
- (2) Model swap on GPU: pre-load, unload cold.
- (3) Weight sharing: distill many task-specific into one MoE / adapter model.
- (4) A/B / champion-challenger co-deploy for comparison.
- (5) Autoscale per model based on QPS.
- Tradeoffs: cold-start latency, memory pressure, tail latency variance.
- Modern: LoRA adapters share base weights across tasks (LLM serving).
Why make inference requests idempotent?
medium- Same request produces same result (or same side effects).
- Enables: (1) safe retry on network / server errors.
- (2) deduplication of accidentally duplicated requests.
- (3) caching by request hash.
- Implementation: (1) client passes idempotency-key header.
- (2) server logs (key, result) → returns cached on repeat.
- (3) side effects (write to DB) also keyed.
- Standard for high-value predictions (payments, credit approvals).
- Stripe pattern.
How to trade model quality vs latency in production?
medium- (1) Distill large model into smaller.
- (2) Ensemble → single model.
- (3) Quantize (fp32 → int8) — usually <1% quality loss.
- (4) Prune (remove low-magnitude weights).
- (5) Two-stage: cheap first, expensive rerank on top-K.
- (6) Adaptive compute: run cheap model, escalate uncertain cases to expensive.
- (7) Cache results for repeated queries.
- (8) Speculative decoding for autoregressive.
- Measure Pareto frontier: for each latency budget, what's best quality?
- Choose based on business SLA.
How do you configure autoscaling for ML serving?
hard- (1) Metric: QPS + GPU util + p95 latency (not just CPU).
- (2) HPA (K8s Horizontal Pod Autoscaler) with custom metrics via Prometheus adapter.
- (3) Scale out threshold + cooldown period (avoid flapping).
- (4) Min instances > 0 for warm capacity (avoid cold starts).
- (5) Max instances = budget ceiling.
- (6) Provisioned min + burst on demand (Cloud Run / Lambda).
- (7) Predictive scaling for known patterns (e.g., 9am weekday).
- Alternative: KEDA event-driven scaling.
How do you serve ML models across multiple regions?
hard- (1) Deploy model to each region: latency to nearest user.
- (2) Global load balancer routes by geography.
- (3) Feature stores per region (or globally-replicated).
- (4) Model registry syncs across regions.
- (5) Consistency: eventually-consistent model versions OK; strong consistency for safety-critical.
- (6) Failover: healthchecks + auto-route to backup region.
- (7) Region-specific compliance (GDPR EU, data residency).
- Multi-cloud (AWS + GCP) for resilience but complex.
Shadow deployment, canary, or A/B test — which do you use when?
medium- They answer different questions, so the sequence matters more than the choice.
- Shadow mode sends real traffic to the new model without using its output, which validates that it runs, that latency is acceptable and that predictions are sane, with zero user risk but no information about business impact.
- Canary sends a small fraction of real traffic to the new model to catch operational failures under genuine load and to limit blast radius, and it is about safety rather than measurement.
- An A/B test with proper randomization and enough exposure is the only one that measures whether the new model is actually better on the metric you care about.
- The mature pattern is shadow, then canary, then A/B, and only then a full rollout.
What does a real rollback plan for a model require?
medium- The previous model artefact still deployable, which means versioned artefacts with their exact dependency set, not just a file in object storage.
- It also requires the previous feature pipeline, because a rollback that feeds the old model new-format features is not a rollback.
- A configuration switch that routes traffic without a code deployment, so recovery takes seconds rather than a release cycle.
- A defined trigger, stated in advance as a metric and threshold, since deciding whether to roll back during an incident wastes the time you do not have.
- And an explicit answer for state already written by the bad model, such as decisions applied to accounts, because reverting the model does not revert its effects.
- Rehearse it, or it does not exist.