What is p-hacking and how do you prevent it?
mediumAnswer
- Running many analyses (test variants, subgroups, transformations) until you find p < 0.05, then reporting only the significant results.
- Inflates false-positive rate dramatically.
- Prevention: (1) pre-register hypotheses + analysis plan; (2) Bonferroni / FDR on all tested variants; (3) separate exploration (open) from confirmation (locked test set + pre-registered stat plan); (4) report all comparisons, not just significant ones.
- Foundation of the replication crisis.
Check yourself — multiple choice
- Not real
- Try many analyses until p < 0.05; prevent via pre-registration, multiple-testing correction, exploration vs confirmation split, full reporting
- Same as effect size
- Just use p-values
P-hacking: multiple analyses → false positives; prevent via pre-reg + correction.
#hypothesis-testing#multiple-testing
Practise Statistics Fundamentals
215 interview questions in this topic.