How do you measure long-term effects when A/B tests are short?

hard

Answer

$(1) Long holdout experiment: keep 1% of traffic unchanged for months \to measure long-term differential.$
$(2) Surrogate metrics: proxy predicting long-term outcomes (predicted LTV, 28-day retention as surrogate for annual retention).$
$(3) Two-stage experiment: short-term A/B + Bayesian imputation of long-term with historical priors.$
$(4) Meta-analyses across many past experiments.$
$Modern practice: combine surrogate + long holdout at big platforms.$

Check yourself — multiple choice

Impossible
Long holdouts (months at 1% traffic) + surrogate metrics (predicted LTV / retention) + Bayesian imputation from historical priors + meta-analysis
Random
Only short-term

Long-term: long holdouts + surrogate metrics + Bayesian meta-analysis.

#ab-testing

215 interview questions in this topic.