Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Sep 1;5(9):e2233946.
doi: 10.1001/jamanetworkopen.2022.33946.

Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review

Deborah Plana  1 , Dennis L Shung  2 , Alyssa A Grimshaw  3 , Anurag Saraf  4 , Joseph J Y Sung  5 , Benjamin H Kann  6
Affiliations

Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review

Deborah Plana et al. JAMA Netw Open. .

Abstract

Importance: Despite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machine learning interventions are being incorporated into clinical trials in health care.

Objective: To systematically examine the design, reporting standards, risk of bias, and inclusivity of RCTs for medical machine learning interventions.

Evidence review: In this systematic review, the Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection online databases were searched and citation chasing was done to find relevant articles published from the inception of each database to October 15, 2021. Search terms for machine learning, clinical decision-making, and RCTs were used. Exclusion criteria included implementation of a non-RCT design, absence of original data, and evaluation of nonclinical interventions. Data were extracted from published articles. Trial characteristics, including primary intervention, demographics, adherence to the CONSORT-AI reporting guideline, and Cochrane risk of bias were analyzed.

Findings: Literature search yielded 19 737 articles, of which 41 RCTs involved a median of 294 participants (range, 17-2488 participants). A total of 16 RCTS (39%) were published in 2021, 21 (51%) were conducted at single sites, and 15 (37%) involved endoscopy. No trials adhered to all CONSORT-AI standards. Common reasons for nonadherence were not assessing poor-quality or unavailable input data (38 trials [93%]), not analyzing performance errors (38 [93%]), and not including a statement regarding code or algorithm availability (37 [90%]). Overall risk of bias was high in 7 trials (17%). Of 11 trials (27%) that reported race and ethnicity data, the median proportion of participants from underrepresented minority groups was 21% (range, 0%-51%).

Conclusions and relevance: This systematic review found that despite the large number of medical machine learning-based algorithms in development, few RCTs for these technologies have been conducted. Among published RCTs, there was high variability in adherence to reporting standards and risk of bias and a lack of participants from underrepresented minority groups. These findings merit attention and should be considered in future RCT design and reporting.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: None reported.

Figures

Figure 1.
Figure 1.. Screening and Selection of Randomized Clinical Trials
AI indicates artificial intelligence. aDatabases and registers included Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection.
Figure 2.
Figure 2.. Characteristics of Randomized Clinical Trials
A total of 41 randomized clinical trials were included in the analysis.,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, Individuals from underrepresented minority groups were participants in 11 clinical trials in which information on participant race and/or ethnicity was reported.,,,,,,,,,, B, Data for 2021 are from January through October 15. D, The other medical specialty category includes anesthesiology, cardiac surgery, emergency medicine, general surgery, gynecology, intensive care, ophthalmology, pulmonology, and radiology.
Figure 3.
Figure 3.. Adherence to Consolidated Standards of Reporting Trials–Artificial Intelligence (CONSORT-AI) Extension Guideline
A total of 41 randomized clinical trials were included in the analysis.,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, The CONSORT-AI extension is an internationally developed consensus document reflecting recommended clinical trial reporting characteristics to ensure transparency and reproducibility.
Figure 4.
Figure 4.. Risk of Bias in Randomized Clinical Trials
A total of 41 randomized clinical trials were included in the analysis.,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, Risk of bias was assessed using the revised Cochrane Risk of Bias, version 2 tool for randomized clinical trials.
See this image and copyright information in PMC

References

    1. Aung YYM, Wong DCS, Ting DSW. The promise of artificial intelligence: a review of the opportunities and challenges of artificial intelligence in healthcare. Br Med Bull. 2021;139(1):4-15. doi:10.1093/bmb/ldab016 - DOI - PubMed
    1. Wang F, Casalino LP, Khullar D. Deep learning in medicine—promise, progress, and challenges. JAMA Intern Med. 2019;179(3):293-294. doi:10.1001/jamainternmed.2018.7117 - DOI - PubMed
    1. Yue W, Wang Z, Chen H, Payne A, Liu X. Machine learning with applications in breast cancer diagnosis and prognosis. Designs. 2018;2(2):13. doi:10.3390/designs2020013 - DOI
    1. Raita Y, Goto T, Faridi MK, Brown DFM, Camargo CA Jr, Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care. 2019;23(1):64. doi:10.1186/s13054-019-2351-7 - DOI - PMC - PubMed
    1. Johnson AEW, Ghassemi MM, Nemati S, Niehaus KE, Clifton DA, Clifford GD. Machine learning and decision support in critical care. Proc IEEE Inst Electr Electron Eng. 2016;104(2):444-466. doi:10.1109/JPROC.2015.2501978 - DOI - PMC - PubMed

Publication types