Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Sep 5;12(9):e061519.
doi: 10.1136/bmjopen-2022-061519.

Quality of reporting of randomised controlled trials of artificial intelligence in healthcare: a systematic review

Rida Shahzad  1 , Bushra Ayub  2 , M A Rehman Siddiqui  3
Affiliations

Quality of reporting of randomised controlled trials of artificial intelligence in healthcare: a systematic review

Rida Shahzad et al. BMJ Open. .

Abstract

Objectives: The aim of this study was to evaluate the quality of reporting of randomised controlled trials (RCTs) of artificial intelligence (AI) in healthcare against Consolidated Standards of Reporting Trials-AI (CONSORT-AI) guidelines.

Design: Systematic review.

Data sources: We searched PubMed and EMBASE databases for studies reported from January 2015 to December 2021.

Eligibility criteria: We included RCTs reported in English that used AI as the intervention. Protocols, conference abstracts, studies on robotics and studies related to medical education were excluded.

Data extraction: The included studies were graded using the CONSORT-AI checklist, comprising 43 items, by two independent graders. The results were tabulated and descriptive statistics were reported.

Results: We screened 1501 potential abstracts, of which 112 full-text articles were reviewed for eligibility. A total of 42 studies were included. The number of participants ranged from 22 to 2352. Only two items of the CONSORT-AI items were fully reported in all studies. Five items were not applicable in more than 85% of the studies. Nineteen per cent (8/42) of the studies did not report more than 50% (21/43) of the CONSORT-AI checklist items.

Conclusions: The quality of reporting of RCTs in AI is suboptimal. As reporting is variable in existing RCTs, caution should be exercised in interpreting the findings of some studies.

Keywords: clinical trials; health informatics; statistics & research methods.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None declared.

Figures

Figure 1
Figure 1
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart. AI, artificial intelligence; RCT, randomised controlled trial.
Figure 2
Figure 2
Yearwise distribution of RCTs in AI. AI, artificial intelligence; RCT, randomised controlled trial.
Figure 3
Figure 3
Percentage of AI RCTs in different countries and specialties. AI, artificial intelligence; RCT, randomised controlled trial.
See this image and copyright information in PMC

References

    1. McKinney SM, Sieniek M, Godbole V, et al. . International evaluation of an AI system for breast cancer screening. Nature 2020;577:89–94. 10.1038/s41586-019-1799-6 - DOI - PubMed
    1. Esteva A, Kuprel B, Novoa RA, et al. . Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017;542:115–8. 10.1038/nature21056 - DOI - PMC - PubMed
    1. Rajpurkar P, Irvin J, Ball RL, et al. . Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 2018;15:e1002686. 10.1371/journal.pmed.1002686 - DOI - PMC - PubMed
    1. Tyler NS, Mosquera-Lopez CM, Wilson LM, et al. . An artificial intelligence decision support system for the management of type 1 diabetes. Nat Metab 2020;2:612–9. 10.1038/s42255-020-0212-y - DOI - PMC - PubMed
    1. Kim H, Goo JM, Lee KH, et al. . Preoperative CT-based deep learning model for predicting disease-free survival in patients with lung adenocarcinomas. Radiology 2020;296:216–24. 10.1148/radiol.2020192764 - DOI - PubMed

Publication types