Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

doi:10.1038/s41467-024-45355-3

. 2024 Feb 22;15(1):1619.

doi: 10.1038/s41467-024-45355-3.

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

Alexander P L Martindale¹, Carrie D Llewellyn², Richard O de Visser², Benjamin Ng^{3

4}, Victoria Ngai⁵, Aditya U Kale^{6

7

8}, Lavinia Ferrante di Ruffano⁹, Robert M Golub¹⁰, Gary S Collins¹¹, David Moher¹², Melissa D McCradden^{13

14

15}, Lauren Oakden-Rayner¹⁶, Samantha Cruz Rivera^{17

18}, Melanie Calvert^{8

17

18

19

20}, Christopher J Kelly²¹, Cecilia S Lee²², Christopher Yau^{23

24}, An-Wen Chan²⁵, Pearse A Keane²⁶, Andrew L Beam^{27

28}, Alastair K Denniston^{6

7

8

17

26}, Xiaoxuan Liu^{29

30

31}

Affiliations

¹ Brighton and Sussex Medical School, Brighton, UK.
² Department of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, UK.
³ Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK.
⁴ Christ Church, University of Oxford, Oxford, UK.
⁵ University College London Medical School, London, UK.
⁶ Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK.
⁷ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
⁸ National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK.
⁹ York Health Economics Consortium, University of York, Birmingham, UK.
¹⁰ Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA.
¹¹ Centre for Statistics in Medicine//UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
¹² Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottowa, Canada.
¹³ Department of Bioethics, The Hospital for Sick Children, Toronto, Canada.
¹⁴ Genetics & Genome Biology Research Program, Peter Gilgan Centre for Research & Learning, Toronto, Canada.
¹⁵ Division of Clinical and Public Health, Dalla Lana School of Public Health, Toronto, Canada.
¹⁶ Australian Institute for Machine Learning, University of Adelaide, Adelaide, Australia.
¹⁷ Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK.
¹⁸ Centre for Patient Reported Outcomes Research (CPROR), Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁹ NIHR Applied Research Collaboration (ARC) West Midlands, University of Birmingham, Birmingham, UK.
²⁰ NIHR Blood and Transplant Research Unit (BTRU) in Precision Transplant and Cellular Therapeutics, University of Birmingham, Birmingham, UK.
²¹ Google Health, London, UK.
²² University of Washington, Seattle, WA, USA.
²³ Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK.
²⁴ Health Data Research UK, London, UK.
²⁵ Department of Medicine, Women's College Hospital. University of Toronto, Toronto, Canada.
²⁶ NIHR Biomedical Research Centre at Moorfields, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK.
²⁷ Department of Epidemiology, Harvard. T.H. Chan School of Public Health, Boston, MA, USA.
²⁸ Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
²⁹ Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK. xliuphone@gmail.com.
³⁰ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK. xliuphone@gmail.com.
³¹ Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK. xliuphone@gmail.com.

PMID: 38388497
PMCID: PMC10883966
DOI: 10.1038/s41467-024-45355-3

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

Alexander P L Martindale et al. Nat Commun. 2024.

. 2024 Feb 22;15(1):1619.

doi: 10.1038/s41467-024-45355-3.

Authors

Affiliations

¹ Brighton and Sussex Medical School, Brighton, UK.
² Department of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, UK.
³ Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK.
⁴ Christ Church, University of Oxford, Oxford, UK.
⁵ University College London Medical School, London, UK.
⁶ Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK.
⁷ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
⁸ National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK.
⁹ York Health Economics Consortium, University of York, Birmingham, UK.
¹⁰ Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA.
¹¹ Centre for Statistics in Medicine//UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
¹² Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottowa, Canada.
¹³ Department of Bioethics, The Hospital for Sick Children, Toronto, Canada.
¹⁴ Genetics & Genome Biology Research Program, Peter Gilgan Centre for Research & Learning, Toronto, Canada.
¹⁵ Division of Clinical and Public Health, Dalla Lana School of Public Health, Toronto, Canada.
¹⁶ Australian Institute for Machine Learning, University of Adelaide, Adelaide, Australia.
¹⁷ Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK.
¹⁸ Centre for Patient Reported Outcomes Research (CPROR), Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁹ NIHR Applied Research Collaboration (ARC) West Midlands, University of Birmingham, Birmingham, UK.
²⁰ NIHR Blood and Transplant Research Unit (BTRU) in Precision Transplant and Cellular Therapeutics, University of Birmingham, Birmingham, UK.
²¹ Google Health, London, UK.
²² University of Washington, Seattle, WA, USA.
²³ Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK.
²⁴ Health Data Research UK, London, UK.
²⁵ Department of Medicine, Women's College Hospital. University of Toronto, Toronto, Canada.
²⁶ NIHR Biomedical Research Centre at Moorfields, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK.
²⁷ Department of Epidemiology, Harvard. T.H. Chan School of Public Health, Boston, MA, USA.
²⁸ Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
²⁹ Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK. xliuphone@gmail.com.
³⁰ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK. xliuphone@gmail.com.
³¹ Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK. xliuphone@gmail.com.

PMID: 38388497
PMCID: PMC10883966
DOI: 10.1038/s41467-024-45355-3

Erratum in

Author Correction: Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines.
Martindale APL, Llewellyn CD, de Visser RO, Ng B, Ngai V, Kale AU, di Ruffano LF, Golub RM, Collins GS, Moher D, McCradden MD, Oakden-Rayner L, Rivera SC, Calvert M, Kelly CJ, Lee CS, Yau C, Chan AW, Keane PA, Beam AL, Denniston AK, Liu X. Martindale APL, et al. Nat Commun. 2024 Jul 29;15(1):6376. doi: 10.1038/s41467-024-49956-w. Nat Commun. 2024. PMID: 39075055 Free PMC article. No abstract available.

Abstract

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77-94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.

PubMed Disclaimer

Conflict of interest statement

Several authors (X.L., D.M., A.D., C.K., L.F.R., C.L., A.W.C., M.C., P.K., G.S.C., R.G., L.O.R., M.M., C.Y., S.C.R., A.B.) were involved in the development of CONSORT-AI. MC receives funding from the NIHR, UK Research and Innovation (UKRI), NIHR BRC, the NIHR Surgical Reconstruction and Microbiology Research Centre, NIHR ARC West Midlands, NIHR Birmingham-Oxford Blood and Transplant Research Unit (BTRU) in Precision Transplant and Cellular Therapeutics, UKSPINE, European Regional Development Fund – Demand Hub and Health Data Research UK at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Innovate UK (part of UKRI), Macmillan Cancer Support, UCB Pharma, GSK and Gilead. M.C. has received personal fees from Astellas, Aparito Ltd, CIS Oncology, Takeda, Merck, Daiichi Sankyo, Glaukos, GSK and the Patient-Centred Outcomes Research Institute (PCORI) outside the submitted work. X.L. and A.D. have received funding from the NHS AI Lab, The Health Foundation, NIHR, NIHR BRC, MHRA and NICE, outside the submitted work. A.D. and M.C. are supported by the NIHR Birmingham Biomedical Research Centre. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. C.K. is an employee of Google, UK. L.F.R. is an employee of York Health Economics Consortium (YHEC). The remaining authors declare no competing interests.

Figures

**Fig. 2. Location heatmap of included studies by country, showing high distribution within China and USA.**
Generated using R Statistical Software (v4.1.1, R Core Team 2021).

**Fig. 3**
Distribution of clinical specialties amongst included RCTs, showing a high prevalence of interventions within gastroenterology.

See this image and copyright information in PMC

Cited by

A framework for human evaluation of large language models in healthcare derived from literature review.
Tam TYC, Sivarajkumar S, Kapoor S, Stolyar AV, Polanska K, McCarthy KR, Osterhoudt H, Wu X, Visweswaran S, Fu S, Mathur P, Cacciamani GE, Sun C, Peng Y, Wang Y. Tam TYC, et al. NPJ Digit Med. 2024 Sep 28;7(1):258. doi: 10.1038/s41746-024-01258-7. NPJ Digit Med. 2024. PMID: 39333376 Free PMC article.
The testing of AI in medicine is a mess. Here's how it should be done.
Lenharo M. Lenharo M. Nature. 2024 Aug;632(8026):722-724. doi: 10.1038/d41586-024-02675-0. Nature. 2024. PMID: 39169244 No abstract available.
Using ChatGPT to Improve the Presentation of Plain Language Summaries of Cochrane Systematic Reviews About Oncology Interventions: Cross-Sectional Study.
Šuto Pavičić J, Marušić A, Buljan I. Šuto Pavičić J, et al. JMIR Cancer. 2025 Mar 19;11:e63347. doi: 10.2196/63347. JMIR Cancer. 2025. PMID: 40106236 Free PMC article.
Effect of artificial intelligence driven therapeutic lifestyle changes (AI-TLC) intervention on health behavior and health among obesity pregnant women in China: a randomized controlled trial protocol.
Wang X, Bai Y, Li W, Yang C, Zhang L, Zhu H, Bao R, Jiang Y, Wang F, Wang H. Wang X, et al. Front Public Health. 2025 May 12;13:1580060. doi: 10.3389/fpubh.2025.1580060. eCollection 2025. Front Public Health. 2025. PMID: 40421363 Free PMC article.
A Practical Guide to Evaluating Artificial Intelligence Imaging Models in Scientific Literature.
McCarthy A, Valenzuela I, Chen RWS, Dagi Glass LR, Thakoor K. McCarthy A, et al. Ophthalmol Sci. 2025 Jun 9;5(6):100847. doi: 10.1016/j.xops.2025.100847. eCollection 2025 Nov-Dec. Ophthalmol Sci. 2025. PMID: 40778360 Free PMC article.

See all "Cited by" articles

References

1. Tyler, N. S. et al. An artificial intelligence decision support system for the management of type 1 diabetes. Nat. Metab.2, 612–619 (2020). - PMC - PubMed
1. McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature577, 89–94 (2020). - PubMed
1. Beaulieu-Jones, B. K. et al. Machine learning for patient risk stratification: standing on, or looking over, the shoulders of clinicians? npj Digit. Med.4, 1–6 (2021). - PMC - PubMed
1. Liu, X. et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit. Health1, e271–e297 (2019). - PubMed
1. Strohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C. & Moors, E. H. M. Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur. Radio.30, 5525–5532 (2020). - PMC - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources

[1] Tyler, N. S. et al. An artificial intelligence decision support system for the management of type 1 diabetes. Nat. Metab.2, 612–619 (2020). - PMC - PubMed

[2] Tyler, N. S. et al. An artificial intelligence decision support system for the management of type 1 diabetes. Nat. Metab.2, 612–619 (2020). - PMC - PubMed

[3] McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature577, 89–94 (2020). - PubMed

[4] McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature577, 89–94 (2020). - PubMed

[5] Beaulieu-Jones, B. K. et al. Machine learning for patient risk stratification: standing on, or looking over, the shoulders of clinicians? npj Digit. Med.4, 1–6 (2021). - PMC - PubMed

[6] Beaulieu-Jones, B. K. et al. Machine learning for patient risk stratification: standing on, or looking over, the shoulders of clinicians? npj Digit. Med.4, 1–6 (2021). - PMC - PubMed

[7] Liu, X. et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit. Health1, e271–e297 (2019). - PubMed

[8] Liu, X. et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit. Health1, e271–e297 (2019). - PubMed

[9] Strohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C. & Moors, E. H. M. Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur. Radio.30, 5525–5532 (2020). - PMC - PubMed

[10] Strohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C. & Moors, E. H. M. Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur. Radio.30, 5525–5532 (2020). - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

Affiliations

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

Authors

Affiliations

Erratum in

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Erratum in

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources