What is Sample Ratio Mismatch (SRM) and why check it?

medium

Answer

$Chi-square test that observed enrollment ratio (say 50/50) matches expected ratio. p < 0.005 \to strong evidence something's wrong with the randomizer (bot traffic, ID mismatch, bug in assignment logic, biased filtering).$
$Any downstream analysis with SRM is invalid — you can't trust the effect estimate.$
$Standard early-warning check in every experimentation platform (Optimizely, Statsig, Eppo).$

Check yourself — multiple choice

Ignore
Chi-square that assignment ratio matches expected; SRM (p < 0.005) → bug in randomizer/logging → invalidates analysis until fixed
Random
Same as p-hacking

SRM: chi-square that ratio matches; SRM → invalid experiment until fixed.

#ab-testing#design

215 interview questions in this topic.