"Magnitude-based inference": a statistical review

Abstract

Purpose: We consider "magnitude-based inference" and its interpretation by examining in detail its use in the problem of comparing two means.

Methods: We extract from the spreadsheets, which are provided to users of the analysis (http://www.sportsci.org/), a precise description of how "magnitude-based inference" is implemented. We compare the implemented version of the method with general descriptions of it and interpret the method in familiar statistical terms.

Results and conclusions: We show that "magnitude-based inference" is not a progressive improvement on modern statistics. The additional probabilities introduced are not directly related to the confidence interval but, rather, are interpretable either as P values for two different nonstandard tests (for different null hypotheses) or as approximate Bayesian calculations, which also lead to a type of test. We also discuss sample size calculations associated with "magnitude-based inference" and show that the substantial reduction in sample sizes claimed for the method (30% of the sample size obtained from standard frequentist calculations) is not justifiable so the sample size calculations should not be used. Rather than using "magnitude-based inference," a better solution is to be realistic about the limitations of the data and use either confidence intervals or a fully Bayesian analysis.

FIGURE 1

Ternary plot of the probabilities p_b, p_h, and 1 − p_b − p_h showing the four regions corresponding to the different possible conclusions “beneficial,” “trivial,” harmful,” and “unclear” when η_b = 0.25 and η_h = 0.05. The threshold values from Table 1 are represented by gray lines. Note that the 0.005 and 0.995 lines are not actually visible because they are very close to the side of the triangle and the vertex of the triangle, respectively; the lines we can see represent the probabilities 0.05, 0.25, 0.75, and 0.95. The gray p_b labels on the left hand edge of the triangle are for the lines running parallel to the right hand side, and the gray p_h labels on the right hand edge of the triangle are for the lines running parallel to the left hand side. The horizontal lines for 1 − p_b − p_h are drawn in but not labeled to reduce clutter. We have also partitioned the triangle into the regions specified in Table 2 using the threshold values η_b = 0.25 and η_h = 0.05 (we use η_h = 0.05 rather than the default 0.005 to make the region visible.) The regions are shaded to make them easier to distinguish. The region labels are written outside the triangle adjacent to the region. The black point represents values of p_b, p_h, and 1 − p_b − p_h (from the example in the spreadsheet), which lead to the conclusion “possibly beneficial.” The cross on the base represents the values of p_b, p_h, and 1 − p_b − p_h when δ = 0, and the curve through the cross and black point shows the effect of changing δ on p_b, p_h, and 1 − p_b − p_h.

FIGURE 2

Plots of the probabilities of finding beneficial, trivial, or harmful effects as functions of δ for four values of (η_b, η_h) when there is no effect. The 10,000 data sets were simulated to have μ₂ − μ₁ = 0, with similar other characteristics to the example data (n₁ = n₂ = 20, σ²₁ = 15², σ²₂ = 11²). The vertical dashed gray line corresponds to δ = 4.418, the value used in our analysis.

FIGURE 3

Ternary plot showing the distribution of 3000 realizations of the triple p_b, p_h, and 1 − p_b − p_h when δ = 4.418. The data were generated in the same way as the data used in Figure 2.