What techniques reduce variance in A/B tests?
hardAnswer
- (1) CUPED: covariate adjustment using a pre-experiment baseline metric — 30-70% variance reduction on retention metrics.
- (2) Stratification: guarantee balanced enrollment by strata (region, device).
- (3) Doubly-robust estimators.
- (4) Ratio metrics with delta-method variance.
- (5) Winsorize extreme values to reduce heavy-tail variance.
- All target the same goal: smaller CIs at the same n → detect smaller effects.
- CUPED is Microsoft / Meta / Netflix standard.
Check yourself — multiple choice
- No way to reduce
- CUPED (baseline covariate adjustment) / stratification / doubly-robust / delta-method / winsorization — all shrink CIs at fixed n
- Only larger n
- Same as p-hacking
Variance reduction: CUPED / stratification / DR / delta-method / winsorization.
#ab-testing#power
Practise Statistics Fundamentals
215 interview questions in this topic.