EasyDeepLearn
Statistics Fundamentals · section 15 of 15

Experimentation & A/B testing

9 interview questions on experimentation & a/b testing, each answered in full. Free to read, no account needed.

What are the pillars of a solid A/B test design?

medium
  • (1) Clear primary metric + guardrail metrics chosen in advance.
  • (2) Sample size / MDE via power analysis.
  • (3) Randomization unit chosen at correct level (user, session, cluster).
  • (4) Sample ratio check (SRC): assignment counts should match target ratio.
  • (5) A/A test as sanity check on randomizer.
  • (6) Pre-registered analysis plan.
  • (7) Fixed test duration + no-peeking rule (or sequential test).
  • (8) Multiple-testing correction if many metrics.
#ab-testing#designPermalink & quiz →

What is Sample Ratio Mismatch (SRM) and why check it?

medium
  • Chi-square test that observed enrollment ratio (say 50/50) matches expected ratio. p < 0.005 → strong evidence something's wrong with the randomizer (bot traffic, ID mismatch, bug in assignment logic, biased filtering).
  • Any downstream analysis with SRM is invalid — you can't trust the effect estimate.
  • Standard early-warning check in every experimentation platform (Optimizely, Statsig, Eppo).
#ab-testing#designPermalink & quiz →

What are guardrail metrics?

easy
  • Secondary metrics you commit to monitor to catch bad tradeoffs: latency, error rate, retention, revenue-per-user, complaint rate, security signals.
  • Test can 'win' on the primary metric but must not degrade guardrails beyond a preset threshold.
  • Standard practice: bidirectional CI on each guardrail must not cross the harm threshold.
  • Prevents 'winning' by cannibalizing another team's metric or long-term health.
#ab-testing#designPermalink & quiz →

OEC — Overall Evaluation Criterion — what is it?

hard
  • A single scalar composite of relevant metrics chosen ex ante as the experiment's success criterion.
  • Simplifies decision-making (one number).
  • Should reflect business value, be sensitive at reasonable sample sizes, and align long-term with company OKRs.
  • Examples: Bing's 'sessions per user', Netflix engagement composite.
  • Cost: hard to design well; teams often mis-weight components → dumb decisions.
  • Modern practice: use OEC + a small guardrail suite rather than OEC alone.
#ab-testing#designPermalink & quiz →

Novelty vs primacy effects in long experiments.

hard
  • Novelty: users react positively to any change initially, then revert → early effect inflated, long-run effect smaller.
  • Primacy: existing users hate change initially, then adapt → early effect deflated, long-run effect larger.
  • Fixes: (1) run experiments long enough (2-4 weeks minimum) to observe stabilization; (2) segment by user tenure (new vs existing); (3) look at the trend of daily effect over time; (4) always plan for ramp-up + steady-state phases in analysis.
#ab-testingPermalink & quiz →

Switchback experiments — when and how?

hard
  • Alternate treatment on/off over time windows across the whole market → each window randomly assigned.
  • Estimator: difference in outcomes between treated and control windows, with cluster-robust or time-block SEs.
  • Uses: full-market experiments where user-level randomization fails (surge pricing at Uber, driver dispatch at DoorDash).
  • Carryover between windows biases the estimate — use burn-in periods within each window and appropriate window length.
#ab-testing#quasi-experimentalPermalink & quiz →

What is the winner's curse in A/B testing?

hard
  • Because we only 'ship' experiments that cross the significance / effect threshold, the observed effect on the winners is a biased over-estimate of the true effect (selection on the outcome).
  • Empirical rule at Microsoft / Bing: shipped effects shrink 10-40% on re-measurement.
  • Fixes: (1) empirical Bayes shrinkage of shipped estimates; (2) look at long-run replicable effects; (3) hold-out samples to re-estimate on unbiased data.
  • Foundation of realistic ROI accounting.
#ab-testingPermalink & quiz →

How do you measure long-term effects when A/B tests are short?

hard
  • (1) Long holdout experiment: keep 1% of traffic unchanged for months → measure long-term differential.
  • (2) Surrogate metrics: proxy predicting long-term outcomes (predicted LTV, 28-day retention as surrogate for annual retention).
  • (3) Two-stage experiment: short-term A/B + Bayesian imputation of long-term with historical priors.
  • (4) Meta-analyses across many past experiments.
  • Modern practice: combine surrogate + long holdout at big platforms.
#ab-testingPermalink & quiz →

How does a company scale to running 1000+ experiments concurrently?

hard
  • (1) Layered assignment (users get one exposure per orthogonal layer) — Google's classic approach.
  • (2) Mutually exclusive groups when interactions are expected.
  • (3) Centralized experimentation platform (Optimizely, Statsig, Eppo, in-house Google Optimize, Meta Deltoid).
  • (4) Metric standardization + guardrails.
  • (5) FDR control across the metric-experiment matrix.
  • (6) A/A tests continuously to monitor platform health.
  • (7) Culture of pre-registered hypotheses + shipping thresholds.
  • Big-tech DS interviews probe this.
#ab-testing#designPermalink & quiz →

Practise Statistics Fundamentals

All 15 sectionsFlashcardsQuiz