Why running many tests is risky
Each test at α = .05 has a 5% chance of a false positive when there is no real effect. Run enough tests and a false positive becomes likely: the chance of at least one is 1 − (1 − α)k. Looking through many comparisons and reporting only what came out significant is one form of p-hacking.
Worked example
Twenty independent tests at α = .05 give a 64.2% chance of at least one false positive, with one false positive expected on average. To hold the overall risk at 5%, Bonferroni tests each at .05 / 20 = .0025, and Šidák at .00256.
What to do about it
- Decide your main outcome and analysis before seeing the data, and preregister them if you can.
- Report every test you ran, not only the significant ones.
- Adjust p-values with the multiple comparisons calculator, which offers Bonferroni, Holm and Benjamini–Hochberg.
- If you check results as data come in, use a sequential test.
Limits of this calculation
The formula assumes independent tests where every null hypothesis is true. Correlated tests produce a somewhat lower risk, though Bonferroni still controls it. Repeatedly checking one growing dataset is a different case: in our simulation of 20,000 null experiments, checking after every 10 participants per group up to 100 gave a false-positive rate of 19.4%.
Frequently asked questions
What is the family-wise error rate?
The probability of at least one false positive across a set of tests. For k independent tests with all nulls true, it equals 1 - (1 - alpha)^k.
How many tests until a false positive is likely?
At alpha = .05, 14 independent tests are enough to pass a 50% chance of at least one false positive.
What is the Bonferroni correction?
Divide alpha by the number of tests. It is simple and valid under dependence, though conservative when there are many tests.
Does this apply to checking results repeatedly?
The risk also grows, but the looks are correlated, so the formula overstates it. Sequential methods are designed for this situation.