For a proper use of frequentist inferential statistics in public health

Abstract

As widely noted in the literature and by international bodies such as the American Statistical Association, severe misinterpretations of P-values, confidence intervals, and statistical significance are sadly common in public health. This scenario poses serious risks concerning terminal decisions such as the approval or rejection of therapies. Cognitive distortions about statistics likely stem from poor teaching in schools and universities, overly simplified interpretations, and - as we suggest - the reckless use of calculation software with predefined standardized procedures. In light of this, we present a framework to recalibrate the role of frequentist-inferential statistics within clinical and epidemiological research. In particular, we stress that statistics is only a set of rules and numbers that make sense only when properly placed within a well-defined scientific context beforehand. Practical examples are discussed for educational purposes. Alongside this, we propose some tools to better evaluate statistical outcomes, such as multiple compatibility or surprisal intervals or tuples of various point hypotheses. Lastly, we emphasize that every conclusion must be informed by different kinds of scientific evidence (e.g., biochemical, clinical, statistical, etc.) and must be based on a careful examination of costs, risks, and benefits.

Keywords: Clinical significance; Confidence intervals; Null hypothesis; Nullism; Statistical compatibility; Statistical significance.

Fig. 1

Welch t-distribution example (df = 9, t* = −2.96). The orange area corresponds to P = 0.016.

Fig. 2

Hazard ratios and 95% compatibility intervals for three different studies (replications).

Fig. 3

Critical values (t_c) for the t-distribution.