A general introduction to adjustment for multiple comparisons

Abstract

In experimental research a scientific conclusion is always drawn from the statistical testing of hypothesis, in which an acceptable cutoff of probability, such as 0.05 or 0.01, is used for decision-making. However, the probability of committing false statistical inferences would considerably increase when more than one hypothesis is simultaneously tested (namely the multiple comparisons), which therefore requires proper adjustment. Although the adjustment for multiple comparisons is proposed to be mandatory in some journals, it still remains difficult to select a proper method suitable for the various experimental properties and study purposes, especially for researchers without good background in statistics. In the present paper, we provide a brief review on mathematical framework, general concepts and common methods of adjustment for multiple comparisons, which is expected to facilitate the understanding and selection of adjustment methods.

Keywords: Multiple comparisons; adjustment; statistical inference.

Figure 1

The increased error rate of multiple comparisons.

Figure 2

Differences of the adjusted P values among various methods. The dashed horizontal line denotes the pre-specified significance level.

Figure 3

Schematic illustration for Holm adjustment.