Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Nov 1;31(11):2730-2739.
doi: 10.1093/jamia/ocae209.

Toward a responsible future: recommendations for AI-enabled clinical decision support

Steven Labkoff  1   2 , Bilikis Oladimeji  3 , Joseph Kannry  4 , Anthony Solomonides  5 , Russell Leftwich  6 , Eileen Koski  7 , Amanda L Joseph  8 , Monica Lopez-Gonzalez  9 , Lee A Fleisher  10 , Kimberly Nolen  11 , Sayon Dutta  12   13   14 , Deborah R Levy  15   16 , Amy Price  17   18 , Paul J Barr  17 , Jonathan D Hron  19   20 , Baihan Lin  21   22 , Gyana Srivastava  2   23 , Nuria Pastor  24 , Unai Sanchez Luque  24 , Tien Thi Thuy Bui  25 , Reva Singh  26 , Tayler Williams  26 , Mark G Weiner  27 , Tristan Naumann  28 , Dean F Sittig  29 , Gretchen Purcell Jackson  30   31 , Yuri Quintana  2   8   14   32
Affiliations

Toward a responsible future: recommendations for AI-enabled clinical decision support

Steven Labkoff et al. J Am Med Inform Assoc. .

Abstract

Background: Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging.

Objectives: This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients.

Materials and methods: In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process.

Results: Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided.

Discussion: AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow.

Conclusions: Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.

Keywords: algorithmic transparency; artificial intelligence; clinical decision support; clinician AI competencies; patient safety.

PubMed Disclaimer

Conflict of interest statement

S.L. is an employee of Quantori, LLC. B.O. is an employee of UnitedHealth Group. R.L. is a full-time employee of InterSystems Corporation. E.K. owns stock in IBM. L.A.F. is a Principal at Rubrum Advising, which advises companies on coverage of new technologies, including AI. K.N. is an employee and shareholder of Pfizer Inc. T.N. is an employee of Microsoft Corporation. D.F.S. is an owner of Informatics Review LLC. G.P.J. is an employee of Intuitive Surgical. G.P.J. owns stock and/or stock options for IBM, Kyndryl, and Intuitive Surgical. G.P.J. was the Past President and Past Board Chair of the American Medical Informatics Association. The authors had no other conflicts or competing interests to disclose.

Figures

Figure 1.
Figure 1.
Human-centered AI development cycle.
See this image and copyright information in PMC

References

    1. Collen MF. Computer medicine: its application today and tomorrow. Minn Med. 1966;49(11):1705-1707. - PubMed
    1. Middleton B, Sittig DF, Wright A.. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform. 2016;25(Suppl 1):S103-S116. 10.15265/IYS-2016-s034 - DOI - PMC - PubMed
    1. Bleich HL. Computer-based consultation: electrolyte and acid-base disorders. Am J Med. 1972;53(3):285-291. - PubMed
    1. Parker RC, Miller RA.. Creation of realistic appearing simulated patient cases using the INTERNIST-1/QMR knowledge base and interrelationship properties of manifestations. Methods Inf Med. 1989;28(4):346-351. - PubMed
    1. Barnett GO, Cimino JJ, Hupp JA, et al.DXPlain: an evolving diagnostic decision-support system. JAMA. 1987;258(1):67-74. - PubMed