Why is multiple testing a problem and how do you correct for it?
mediumAnswer
- Running many independent tests at will produce many false positives by chance alone (5% per test).
- Corrections: Bonferroni (divide alpha by number of tests — very conservative), Holm-Bonferroni (stepwise, better power), or control the False Discovery Rate with Benjamini-Hochberg (best for discovery-style work with many tests).
Check yourself — multiple choice
- Running 100 tests at gives at most 5 false positives
- Benjamini-Hochberg controls the false discovery rate
- Bonferroni increases the false positive rate
- Multiple testing is not an issue in ML experiments
BH is the standard FDR-controlling procedure.
#hypothesis-testing#multiple-testing
Practise Statistics Fundamentals
215 interview questions in this topic.