The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement

Abstract

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Fig 1. Flowchart of how the final full-text randomized clinical trials were obtained.

1. RCT, randomized controlled trial.

Fig 2

Number of RCTs included over time (A) and the corresponding average number of authors (B), the number of countries involved (C), and the number of institutions involved (D). The indicated stratum range is up to but not including the last year. Raw data are available via DOI 10.5281/zenodo.4362238. RCT, randomized controlled trial.

Fig 3

Risk of bias due to inadequate allocation concealment (A), random sequence generation bias (B), the bias in blinding of patients and personnel (people) (C), the bias in blinding of outcome assessment (D), RCT registration (E), and reporting of the CONSORT Statement (F) for all RCTs plotted over time. The indicated stratum range is up to but not including the last year. Raw data are available via DOI 10.5281/zenodo.4362238. RCT, randomized controlled trial.

Fig 4

Risk of bias in allocation concealment (A), the bias in randomization (B), the bias in blinding of patients and personnel (people) (C), the bias in blinding of outcome assessment (D), RCT registration (E), and CONSORT Statement reporting (F) plotted over time for RCTs published in journals with JIF >10 and journals with JIF <10. The indicated stratum range is up to but not including the last year. Raw data are available via DOI 10.5281/zenodo.4362238. JIF, journal impact factor; RCT, randomized controlled trial.