Statistics Fundamentals · section 8 of 15

Power, effect size & multiple testing

5 interview questions on power, effect size & multiple testing, each answered in full. Free to read, no account needed.

What techniques reduce variance in A/B tests?

hard

$(1) CUPED: covariate adjustment using a pre-experiment baseline metric — 30-70% variance reduction on retention metrics.$
$(2) Stratification: guarantee balanced enrollment by strata (region, device).$
$(3) Doubly-robust estimators.$
$(4) Ratio metrics with delta-method variance.$
$(5) Winsorize extreme values to reduce heavy-tail variance.$
$All target the same goal: smaller CIs at the same n \to detect smaller effects.$
$CUPED is Microsoft / Meta / Netflix standard.$

hard

$CUPED (Controlled-experiment Using Pre-Experiment Data, Deng et al. 2013): use a pre-experiment covariate X (e.g. same metric, 30 days pre-treatment) to explain part of Y's variance.$
$Adjusted metric: Y' = Y - θ * (X - E[X]), where θ = Cov(Y, X)/Var(X).$
$\operatorname{Var}(Y)\; = \;\operatorname{Var}(Y)\; \cdot \;(1\; - \;\operatorname{Corr}(Y, \;X)^{2})$
$Typical 30-70% variance reduction on retention / spend metrics \to 3-5x fewer users for same MDE.$
$Standard at Microsoft, Meta, Netflix, Booking.$

medium

$Because the false positive rate is not the nominal 5% under repeated looks; every additional peek is another chance for random fluctuation to cross the threshold, and with daily checks over a few weeks the real error rate rises to something like 20 to 30%.$
$Fixed-horizon tests assume exactly one analysis at a predetermined sample size, and stopping on the first significant result systematically selects the fluctuations that happened to be extreme.$
$The honest options are to fix the sample size in advance from a power calculation and look only once, or to use a method designed for continuous monitoring: a sequential test with alpha spending such as a group sequential design, or always-valid confidence sequences.$
$If you must peek for operational safety, peek at guardrails and not at the primary metric.$

#ab-testing#multiple-testingPermalink & quiz →

medium

$Four things, and the hard one is not statistical.$
$The baseline rate of the metric, its variance, which for a proportion follows from the baseline but for a revenue metric must be estimated and is usually much larger than people expect.$
$The minimum detectable effect, meaning the smallest improvement that would actually change a decision, which is a business judgement and the input most often chosen to make the number look acceptable.$
$Then the significance level and power, conventionally 5% and 80%, where power is the probability of detecting the effect if it is real.$
$Sample size scales inversely with the square of the effect size, so halving the detectable effect quadruples the sample, which is why heavy-tailed revenue metrics often cannot be tested directly and are replaced by a capped or binary proxy.$

medium

$Treat it as expected noise until proven otherwise, because with 20 independent tests at the 5% level you expect about one false positive even when nothing changed.$
$The structural fix is to declare a single primary metric before the experiment, with the rest labelled as secondary or guardrail, which turns a fishing expedition into a hypothesis test.$
$If several metrics genuinely need decision authority, control the error rate explicitly: Bonferroni is conservative and fine for a handful, and the Benjamini-Hochberg procedure controls the false discovery rate and is the better choice when you have many.$
$For the metric that fired, ask whether it moved in a direction consistent with a plausible mechanism and with the other metrics, and if it is isolated and surprising, the correct action is to replicate rather than to launch.$

#multiple-testing#ab-testingPermalink & quiz →