Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Apr 7:13:830345.
doi: 10.3389/fpsyg.2022.830345. eCollection 2022.

Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

Stephanie Tulk Jesso  1   2 , Aisling Kelliher  3 , Harsh Sanghavi  4 , Thomas Martin  2   5 , Sarah Henrickson Parker  1   6
Affiliations

Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

Stephanie Tulk Jesso et al. Front Psychol. .

Abstract

The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the "gold standard" of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users' needs and feedback in the design process.

Keywords: artificial intelligence (AI); clinical AI; clinician; evaluation; healthcare; human-centered design; machine learning.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

FIGURE 1
FIGURE 1
Systematic review process and numbers of articles included and excluded. Articles were first identified through online databases, then pre-processed using python scripts. Next, all articles were manually reviewed by title and abstract prior to reviewing full articles to evaluate which articles were included.
FIGURE 2
FIGURE 2
Methods used within included articles and counts.
FIGURE 3
FIGURE 3
Total number of participants included in studies.
See this image and copyright information in PMC

References

    1. Arndt B. G., Beasley J. W., Watkinson M. D., Temte J. L., Tuan W. J., Sinsky C. A., et al. (2017). Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann. Fam. Med. 15 419–426. 10.1370/afm.2121 - DOI - PMC - PubMed
    1. Asan O., Choudhury A. (2021). Research trends in artificial intelligence applications in human factors health care: mapping review. JMIR Hum. Factors 8:e28236. 10.2196/28236 - DOI - PMC - PubMed
    1. Ash M., Petro J., Rab S. (2019). How AI in the exam room could reduce physician burnout. Harv. Bus. Rev. 2–5. Available online at: https://hbr.org/2019/11/how-ai-in-the-exam-room-could-reduce-physician-b...
    1. Bates D. W., Auerbach A., Schulam P., Wright A., Saria S. (2020). Reporting and implementing interventions involving machine learning and artificial intelligence. Ann. Intern. Med. 172 S137–S144. 10.7326/M19-0872 - DOI - PubMed
    1. Baxter S. L., Bass J. S., Sitapati A. M. (2020). Barriers to implementing an artificial intelligence model for unplanned readmissions. ACI Open 4 e108–e113. 10.1055/S-0040-1716748 - DOI - PMC - PubMed

Publication types