Why is checking an A/B test daily and stopping when it turns significant wrong?

medium

Answer

$Because the false positive rate is not the nominal 5% under repeated looks; every additional peek is another chance for random fluctuation to cross the threshold, and with daily checks over a few weeks the real error rate rises to something like 20 to 30%.$
$Fixed-horizon tests assume exactly one analysis at a predetermined sample size, and stopping on the first significant result systematically selects the fluctuations that happened to be extreme.$
$The honest options are to fix the sample size in advance from a power calculation and look only once, or to use a method designed for continuous monitoring: a sequential test with alpha spending such as a group sequential design, or always-valid confidence sequences.$
$If you must peek for operational safety, peek at guardrails and not at the primary metric.$

Check yourself — multiple choice

It is fine, more data is always better
Repeated looks inflate the false positive rate well above 5% because you stop on extreme fluctuations — use a fixed horizon with one analysis, or a sequential test with alpha spending
It only reduces power
Peeking has no effect on the p-value

Optional stopping inflates type I error; sequential designs exist precisely to allow monitoring.

#ab-testing#multiple-testing

215 interview questions in this topic.