Multiple testing: choosing Bonferroni, Holm or FDR control

A false positive here means rejecting a null hypothesis that is actually true. The error level used for an individual comparison is not the same as the error level when all comparisons are considered together. NIST: multiple comparisons

Hypothetical calculation: Suppose all null hypotheses are true, the tests are independent, and no adjustment is applied. If each test has an actual false-rejection probability of exactly 5%, the probability of at least one false rejection across m tests is 1 − (1 − 0.05)^m.

For example, avoiding every false rejection across 20 tests requires all 20 tests to make the correct decision: 0.95^20 ≈ 0.3585. Its complement is approximately 64.2%. This does not mean that every study with 20 tests has a 64.2% probability of being wrong. The calculation does not apply directly when the truth of the null hypotheses, dependence between tests, or actual error probabilities differ. These are teaching probabilities calculated under stated conditions, not research findings.

Imagine an educational study measuring an immediate post-test score, a test score one month later, task completion time, and self-confidence. If the researcher will declare the teaching intervention successful whenever any of these four outcomes is statistically significant, they should explain how the four tests contribute to that decision. Selecting only small p-values and defining a narrower family afterward hides the original selection problem.

You could add the following record to your analysis plan:

This record format is an AcademicFix working suggestion; it does not mean every study must have a single family of tests. Justify separate families through the research questions and decision structure. Prespecified comparisons and comparisons chosen after seeing the results do not have the same planning status. NIST: choosing comparisons

Calling an analysis exploratory does not, by itself, remove the multiple-testing problem.

For FDR, the false-rejection proportion is defined as zero when no hypotheses are rejected. Controlling FDR at 5% does not mean that exactly 5% of a single study’s findings will be wrong, or that each finding has a 5% probability of being wrong. The original Benjamini–Hochberg paper establishes control under independence; that result does not automatically cover every dependence structure. Benjamini and Hochberg, 1995, pp. 291 and 293

Bonferroni and Holm provide family-wise control with valid individual p-values; Holm does not require independence between tests. Benjamini–Yekutieli is another option for FDR control under general dependence and can be more conservative. Do not choose a method because it happens to produce more favourable results. Official R p.adjust documentation

The following p-values are entirely hypothetical. All four tests belong to the same prespecified family. Bonferroni multiplies the raw p-values by 4 and caps each result at 1. Holm uses decreasing multipliers on the ordered values and ensures that the adjusted values do not decrease along that order. R p.adjust

Three raw values are below 0.05; after either adjustment, only A remains below that level. This does not establish that B and C have no effect. Likewise, A’s small p-value does not establish that its effect is large or important. Interpret the effect estimate, uncertainty, and study design separately. ASA principles on p-values

To reproduce the calculation in R:

p <- c(A = .003, B = .018, C = .041, D = .120)
cbind(raw = p,
      bonferroni = p.adjust(p, method = "bonferroni"),
      holm = p.adjust(p, method = "holm"))

This code reproduces the example calculation. It does not recommend this hypothetical family as the right analysis plan for your study.

Hypothetical reporting example: “The four prespecified outcomes were treated as one family of tests. Holm’s procedure was used to control the family-wise probability of a false rejection at 0.05. Raw and adjusted p-values were reported together; subgroup analyses added later were reported as exploratory.” Adapt this paragraph to describe only what you actually did.