Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Nov 19;22(11):e22421.
doi: 10.2196/22421.

Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study

Sahil Sandhu  1 , Anthony L Lin  2 , Nathan Brajer  2 , Jessica Sperling  3 , William Ratliff  4 , Armando D Bedoya  5 , Suresh Balu  4 , Cara O'Brien  6 , Mark P Sendak  4
Affiliations

Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study

Sahil Sandhu et al. J Med Internet Res. .

Abstract

Background: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care.

Objective: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows.

Methods: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data.

Results: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes.

Conclusions: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.

Keywords: emergency medicine; hospital rapid response team; machine learning; qualitative research; sepsis.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: MS, WR, AB, NB, and CO are named inventors of the Sepsis Watch deep learning model, which was licensed from Duke University by Cohere Med Inc. These authors do not hold any equity in Cohere Med Inc. No other authors have relevant financial disclosures.

Figures

Figure 1
Figure 1
Sepsis Watch workflow. ED: emergency department; RRT: rapid response team.
See this image and copyright information in PMC

References

    1. Rhee C, Jones TM, Hamad Y, Pande A, Varon J, O'Brien C, Anderson DJ, Warren DK, Dantes RB, Epstein L, Klompas M, Centers for Disease Control and Prevention (CDC) Prevention Epicenters Program Prevalence, underlying causes, and preventability of sepsis-associated mortality in US acute care hospitals. JAMA Netw Open. 2019 Feb 1;2(2):e187571. doi: 10.1001/jamanetworkopen.2018.7571. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/10.1001/jam... - DOI - PMC - PubMed
    1. Rhee C, Dantes R, Epstein L, Murphy DJ, Seymour CW, Iwashyna TJ, Kadri SS, Angus DC, Danner RL, Fiore AE, Jernigan JA, Martin GS, Septimus E, Warren DK, Karcz A, Chan C, Menchaca JT, Wang R, Gruber S, Klompas M, CDC Prevention Epicenter Program Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. J Am Med Assoc. 2017 Oct 3;318(13):1241–9. doi: 10.1001/jama.2017.13836. http://europepmc.org/abstract/MED/28903154 - DOI - PMC - PubMed
    1. Torio CM, Moore BJ. National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2013: Statistical Brief #204. Healthcare Cost and Utilization Project (HCUP) [2020-11-13]. https://europepmc.org/abstract/med/27359025.
    1. Islam MM, Nasrin T, Walther BA, Wu C, Yang H, Li Y. Prediction of sepsis patients using machine learning approach: a meta-analysis. Comput Methods Programs Biomed. 2019 Mar;170:1–9. doi: 10.1016/j.cmpb.2018.12.027. - DOI - PubMed
    1. Fleuren LM, Klausch TL, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, Swart EL, Girbes AR, Thoral P, Ercole A, Hoogendoorn M, Elbers PW. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020 Mar;46(3):383–400. doi: 10.1007/s00134-019-05872-y. http://europepmc.org/abstract/MED/31965266 - DOI - PMC - PubMed

Publication types

LinkOut - more resources