Why is peeking at running A/B tests dangerous?
hardAnswer
- Peeking = repeatedly running a test and stopping when p < 0.05 → hugely inflates false-positive rate (up to ~40% for one-sided at α=0.05).
- Fixes: (1) commit to sample size ex ante and don't peek.
- (2) Sequential testing: mSPRT, α-spending (Pocock, O'Brien-Fleming), Bayesian sequential decisions.
- (3) Always-valid inference (Howard & Ramdas): give up 10-20% efficiency for the freedom to check any time.
- Optimizely / Statsig / Eppo implement always-valid or sequential correctly.
Check yourself — multiple choice
- Safe to peek
- Peeking inflates FPR up to ~40%; fixes: pre-committed n, sequential (mSPRT / α-spending), always-valid inference
- Only for regression
- Same as multiple testing
Peeking inflates FPR; use pre-committed n, sequential, or always-valid inference.
#hypothesis-testing#ab-testing
Practise Statistics Fundamentals
215 interview questions in this topic.