How does a company scale to running 1000+ experiments concurrently?

hard

Answer

$(1) Layered assignment (users get one exposure per orthogonal layer) — Google's classic approach.$
$(2) Mutually exclusive groups when interactions are expected.$
$(3) Centralized experimentation platform (Optimizely, Statsig, Eppo, in-house Google Optimize, Meta Deltoid).$
$(4) Metric standardization + guardrails.$
$(5) FDR control across the metric-experiment matrix.$
$(6) A/A tests continuously to monitor platform health.$
$(7) Culture of pre-registered hypotheses + shipping thresholds.$
$Big-tech DS interviews probe this.$

Check yourself — multiple choice

Impossible
Layered assignment (orthogonal layers) + mutually exclusive groups when needed + centralized platform + metric standardization + FDR + A/A monitoring + pre-reg
Random
Only sequential

Scale experimentation: orthogonal layers + mutex + platform + FDR + A/A + pre-reg.

#ab-testing#design

215 interview questions in this topic.