Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Feb 5:27:e67485.
doi: 10.2196/67485.

AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis

Christine Jacob  1 , Noé Brasier  2 , Emanuele Laurenzi  1 , Sabina Heuss #  1 , Stavroula-Georgia Mougiakakou #  3   4 , Arzu Cöltekin #  1 , Marc K Peter #  1
Affiliations

AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis

Christine Jacob et al. J Med Internet Res. .

Abstract

Background: Artificial intelligence (AI) has the potential to revolutionize health care by enhancing both clinical outcomes and operational efficiency. However, its clinical adoption has been slower than anticipated, largely due to the absence of comprehensive evaluation frameworks. Existing frameworks remain insufficient and tend to emphasize technical metrics such as accuracy and validation, while overlooking critical real-world factors such as clinical impact, integration, and economic sustainability. This narrow focus prevents AI tools from being effectively implemented, limiting their broader impact and long-term viability in clinical practice.

Objective: This study aimed to create a framework for assessing AI in health care, extending beyond technical metrics to incorporate social and organizational dimensions. The framework was developed by systematically reviewing, analyzing, and synthesizing the evaluation criteria necessary for successful implementation, focusing on the long-term real-world impact of AI in clinical practice.

Methods: A search was performed in July 2024 across the PubMed, Cochrane, Scopus, and IEEE Xplore databases to identify relevant studies published in English between January 2019 and mid-July 2024, yielding 3528 results, among which 44 studies met the inclusion criteria. The systematic review followed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews. Data were analyzed using NVivo through thematic analysis and narrative synthesis to identify key emergent themes in the studies.

Results: By synthesizing the included studies, we developed a framework that goes beyond the traditional focus on technical metrics or study-level methodologies. It integrates clinical context and real-world implementation factors, offering a more comprehensive approach to evaluating AI tools. With our focus on assessing the long-term real-world impact of AI technologies in health care, we named the framework AI for IMPACTS. The criteria are organized into seven key clusters, each corresponding to a letter in the acronym: (1) I-integration, interoperability, and workflow; (2) M-monitoring, governance, and accountability; (3) P-performance and quality metrics; (4) A-acceptability, trust, and training; (5) C-cost and economic evaluation; (6) T-technological safety and transparency; and (7) S-scalability and impact. These are further broken down into 28 specific subcriteria.

Conclusions: The AI for IMPACTS framework offers a holistic approach to evaluate the long-term real-world impact of AI tools in the heterogeneous and challenging health care context and lays the groundwork for further validation through expert consensus and testing of the framework in real-world health care settings. It is important to emphasize that multidisciplinary expertise is essential for assessment, yet many assessors lack the necessary training. In addition, traditional evaluation methods struggle to keep pace with AI's rapid development. To ensure successful AI integration, flexible, fast-tracked assessment processes and proper assessor training are needed to maintain rigorous standards while adapting to AI's dynamic evolution.

Trial registration: reviewregistry1859; https://tinyurl.com/ysn2d7sh.

Keywords: adoption; artificial intelligence; assessment; clinical practice; clinician; eHealth; efficiency; health technology assessment; implementation.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: CJ is an editorial board member of JMIR Human Factors at the time of this publication.

Figures

Figure 1
Figure 1
Study selection flow diagram based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. AI: artificial intelligence.
Figure 2
Figure 2
Visual overview of the aggregated assessment criteria, organized into clusters and subcriteria. AI: artificial intelligence.
Figure 3
Figure 3
AI for IMPACTS: a comprehensive framework for evaluating the long-term real-world impacts of artificial intelligence (AI)–powered clinician tools.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, Cui C, Corrado G, Thrun S, Dean J. A guide to deep learning in healthcare. Nat Med. 2019 Jan 7;25(1):24–9. doi: 10.1038/s41591-018-0316-z.10.1038/s41591-018-0316-z - DOI - PubMed
    1. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020;3:17. doi: 10.1038/s41746-020-0221-y. https://doi.org/10.1038/s41746-020-0221-y 221 - DOI - DOI - PMC - PubMed
    1. Pressman SM, Borna S, Gomez-Cabello CA, Haider SA, Haider CR, Forte AJ. Clinical and surgical applications of large language models: a systematic review. J Clin Med. 2024 May 22;13(11):3041. doi: 10.3390/jcm13113041. https://www.mdpi.com/resolver?pii=jcm13113041 jcm13113041 - DOI - PMC - PubMed
    1. Xu C, Solomon SA, Gao W. Artificial intelligence-powered electronic skin. Nat Mach Intell. 2023 Dec 18;5(12):1344–55. doi: 10.1038/s42256-023-00760-z. https://europepmc.org/abstract/MED/38370145 - DOI - PMC - PubMed
    1. Muehlematter UJ, Daniore P, Vokinger KN. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015-20): a comparative analysis. Lancet Digit Health. 2021 Mar;3(3):e195–203. doi: 10.1016/S2589-7500(20)30292-2. https://linkinghub.elsevier.com/retrieve/pii/S2589-7500(20)30292-2 S2589-7500(20)30292-2 - DOI - PubMed

Publication types