Statistical inference and effect measures in abstracts of randomized controlled trials, 1975-2021. A systematic review

Abstract

Objective: To examine the time trend of statistical inference, statistical reporting style of results, and effect measures from the abstracts of randomized controlled trials (RCTs).

Study desgin and settings: We downloaded 385,867 PubMed abstracts of RCTs from 1975 to 2021. We used text-mining to detect reporting of statistical inference (p-values, confidence intervals, significance terminology), statistical reporting style of results, and effect measures for binary outcomes, including time-to-event measures. We validated the text mining algorithms by random samples of abstracts.

Results: A total of 320 676 abstracts contained statistical inference. The percentage of abstracts including statistical inference increased from 65% (1975) to 87% (2006) and then decreased slightly. From 1975 to 1990, the sole reporting of language regarding statistical significance was predominant. Since 1990, reporting of p-values without confidence intervals has been the most common reporting style. Reporting of confidence intervals increased from 0.5% (1975) to 29% (2021). The two most common effect measures for binary outcomes were hazard ratios and odds ratios. Number needed to treat and number needed to harm are reported in less than 5% of abstracts with binary endpoints.

Conclusions: Reporting of statistical inference in abstracts of RCTs has increased over time. Increasingly, p-values and confidence intervals are reported rather than just mentioning the presence of "statistical significance". The reporting of odds ratios comes with the liability that the untrained reader will interpret them as risk ratios, which is often not justified, especially in RCTs.

Keywords: Confidence intervals; Randomized controlled trials; Statistics; Statistics and numerical data.

Fig. 1

Annual number of abstracts of randomized controlled trials and percentage of abstracts containing statistical inference, 1975–2021 red graph: estimated proportion of abstracts on RCTs that include statistical inference, by year (LOESS with smooth = 0.2) with 95% confidence interval bands; black dots: observed proportion, by year; dashed line: total number of abstracts of RCTs per year

Fig. 2

Estimated time trends, 1975–2021, of the statistical reporting style in abstracts on randomized controlled trials (non-hierarchical) Blue indicates reporting of confidence intervals; green indicates reporting of numerical p-values or p thresholds, (e.g. p < 0.01); red indicates reporting of significance language. All trend lines are LOESS smoothed with inverse-variance weighting

Fig. 3

Flexibly estimated time trends 1975–2021 of the statistical reporting style in abstracts on randomized controlled trials (hierarchical) blue graphs indicate reporting of confidence intervals; green graphs indicate reporting of numerical p-values or p thresholds (e.g. p < 0.01) without confidence intervals; red graphs indicate reporting of significance without confidence intervals and p-values; all trend lines are LOESS smoothed with inverse-variance weighting