Significance testing as perverse probabilistic reasoning

Abstract

Truth claims in the medical literature rely heavily on statistical significance testing. Unfortunately, most physicians misunderstand the underlying probabilistic logic of significance tests and consequently often misinterpret their results. This near-universal misunderstanding is highlighted by means of a simple quiz which we administered to 246 physicians at two major academic hospitals, on which the proportion of incorrect responses exceeded 90%. A solid understanding of the fundamental concepts of probability theory is becoming essential to the rational interpretation of medical information. This essay provides a technically sound review of these concepts that is accessible to a medical audience. We also briefly review the debate in the cognitive sciences regarding physicians' aptitude for probabilistic inference.

Figure 1

Reference population of patients with appendicitis and fever, showing the result of conditioning on the presence of fever.

Figure 2

Effects on posterior probability of changes in sensitivity, while holding prior probability and false positive rate constant.

Figure 3

Effects on posterior probability of changes in false positive rate, while holding prior probability and sensitivity constant.

Figure 4

Effects on posterior probability of changes in prior probability, while holding sensitivity and false positive rate constant.

Figure 5

Illustration of how the posterior probability depends on the three parameters of Bayes' rule. Each plot shows two curves for the posterior probability as a function of one of the three parameters (with the remaining two parameters held constant) chosen from among one of two sets of values for(Pr(A), Pr(F|A), $Pr(F|\overline{A})$), either (0.3, 0.95, 0.05) or (0.1, 0.7, 0.15).

Figure 6

Distribution of systolic blood pressures for a population of healthy 60-69 year old males (from data in [125]). The value SBP = 138.6 mmHg has a P-value of 0.05, equal to the shaded area under the curve.

Figure 7

ROC curve for the coin flipping experiment with n = 10, H₀ : Pr(Heads) = 0.5 vs.H₁ : Pr(Heads) = 0.7. The curve is generated by varying a threshold between 0 (corresponding to the point (1, 1)) and 10 (corresponding to the point (0, 0)).

Figure 8

Bayesian P-value nomogram for a hypothetical hypothesis testing problem. This nomogram is calculated assuming normal distributions for the null hypothesis H₀ and the alternative hypothesis H₁, with variance equal to one, and means differing by a value of two.