When possible, report a Fisher-exact P value and display its underlying null randomization distribution

Abstract

In randomized experiments, Fisher-exact P values are available and should be used to help evaluate results rather than the more commonly reported asymptotic P values. One reason is that using the latter can effectively alter the question being addressed by including irrelevant distributional assumptions. The Fisherian statistical framework, proposed in 1925, calculates a P value in a randomized experiment by using the actual randomization procedure that led to the observed data. Here, we illustrate this Fisherian framework in a crossover randomized experiment. First, we consider the first period of the experiment and analyze its data as a completely randomized experiment, ignoring the second period; then, we consider both periods. For each analysis, we focus on 10 outcomes that illustrate important differences between the asymptotic and Fisher tests for the null hypothesis of no ozone effect. For some outcomes, the traditional P value based on the approximating asymptotic Student's t distribution substantially subceeded the minimum attainable Fisher-exact P value. For the other outcomes, the Fisher-exact null randomization distribution substantially differed from the bell-shaped one assumed by the asymptotic t test. Our conclusions: When researchers choose to report P values in randomized experiments, 1) Fisher-exact P values should be used, especially in studies with small sample sizes, and 2) the shape of the actual null randomization distribution should be examined for the recondite scientific insights it may reveal.

Keywords: Fisher-exact P values; asymptotic P values; crossover randomized experiments; randomization-based inference; sensitivity analyses.

Fig. 1.

Exact (Upper) and approximating (Lower) null randomization distributions for 10 epigenetic outcome variables, labeled A, B, C, D, E, F, G, H, I, J, as explicated by the rows of Table 1; the test statistic being used is T_Welch, where for each variable the vertical red line indicates the actual observed value of T_Welch.

Fig. 2.

Exact (Upper) and approximating (Lower) null randomization distributions for 10 epigenetic outcome variables, labeled A, B, C, D, E, F, G, H, I, J, as explicated by the rows of Table 2; the test statistic being used is T_paired, where for each variable the vertical red line indicates the actual observed value of T_paired.

Fig. 3.

Asymptotic P values vs. Fisher-exact P values in the epigenomic study (484,531 CpG sites). Legend: (Top, A–C) First period; (Bottom, D–F) Crossover experiment. Left, range = [0,1]; Middle, Fisher-exact P values < 0.05; Right, Fisher-exact P values < 0.001. Red line: P value = P value_min. Blue line: 45° line. (A) Range of asymptotic P values: [0.0000034; 0.9999996]; range of Fisher-exact P values: [1/19,448 = 0.0000514; 1]; 52% of the asymptotic P values are less than the Fisher-exact P values. (B) Range of asymptotic P values: [0.0000034; 0.1653300]; range of Fisher-exact P values: [1/19,448 = 0.0000514; 0.0499794]; 60% of the asymptotic P values are less than the Fisher-exact P values. (C) Range of asymptotic P values: [0.0000034; 0.1598584]; range of Fisher-exact P values: [1/19,448 = 0.0000514; 0.0009770]; 36% of the asymptotic P values are less than the Fisher-exact P values. (D) Range of asymptotic P values: [0.0000011; 0.9999806]; range of Fisher-exact P values: [1/19,448 = 0.0000514; 1]; 63% of the asymptotic P values are less than the Fisher-exact P values. (E) Range of asymptotic P values: [0.0000011; 0.2311926]; range of Fisher-exact P values: [1/19,448 = 0.0000514; 0.0499794]; 68% of the asymptotic P values are less than the Fisher-exact P values. (F) Range of asymptotic P values: [0.0000011; 0.0551901]; range of Fisher-exact P values: [1/19,448 = 0.0000514; 0.0009770]; 38% of the asymptotic P values are less than the Fisher-exact P values.