How does CUPED reduce variance in A/B tests?

hard

Answer

$CUPED (Controlled-experiment Using Pre-Experiment Data, Deng et al. 2013): use a pre-experiment covariate X (e.g. same metric, 30 days pre-treatment) to explain part of Y's variance.$
$Adjusted metric: Y' = Y - θ * (X - E[X]), where θ = Cov(Y, X)/Var(X).$
$\operatorname{Var}(Y)\; = \;\operatorname{Var}(Y)\; \cdot \;(1\; - \;\operatorname{Corr}(Y, \;X)^{2})$
$Typical 30-70% variance reduction on retention / spend metrics \to 3-5x fewer users for same MDE.$
$Standard at Microsoft, Meta, Netflix, Booking.$

Check yourself — multiple choice

Random
Adjust metric by pre-experiment covariate X: Y' = Y - θ(X - E[X]) with θ = Cov(Y,X)/Var(X); Var reduced by $(1\; - \;{\rho}^{2})$ → 30-70% at scale; industry standard
Only Bayesian
Not real

CUPED: Y' = Y - θ(X-E[X]); variance × $(1 - {\rho}^{2})$ ; 30-70% at scale.

#ab-testing#power

215 interview questions in this topic.