Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study

doi:10.1136/bmj-2021-068576

Multicenter Study

. 2022 Feb 17:376:e068576.

doi: 10.1136/bmj-2021-068576.

Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study

Fahad Kamran^{1

2}, Shengpu Tang^{1

2}, Erkin Otles^{3

4}, Dustin S McEvoy⁵, Sameh N Saleh^{6

7}, Jen Gong⁸, Benjamin Y Li^{1

4}, Sayon Dutta^{5

9}, Xinran Liu¹⁰, Richard J Medford^{6

7}, Thomas S Valley^{11

12}, Lauren R West¹³, Karandeep Singh^{11

14}, Seth Blumberg^{10

15}, John P Donnelly^{11

14}, Erica S Shenoy^{13

16

17}, John Z Ayanian^{11

12}, Brahmajee K Nallamothu^{11

12}, Michael W Sjoding^{11

12

18}, Jenna Wiens^{19

11

18}

Affiliations

¹ Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, MI 48109, USA.
² Joint first authors.
³ Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor, MI, USA.
⁴ Medical Scientist Training Program, University of Michigan Medical School, Ann Arbor, MI, USA.
⁵ Mass General Brigham Digital Health eCare, Somerville, MA, USA.
⁶ Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX, USA.
⁷ Clinical Informatics Center, University of Texas Southwestern Medical Center, Dallas, TX, USA.
⁸ Center for Clinical Informatics and Improvement Research, University of California, San Francisco, CA, USA.
⁹ Department of Emergency Medicine, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA.
¹⁰ Division of Hospital Medicine, University of California, San Francisco, San Francisco, CA, USA.
¹¹ Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor, MI, USA.
¹² Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, MI, USA.
¹³ Infection Control Unit, Massachusetts General Hospital, Boston, MA, USA.
¹⁴ Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA.
¹⁵ Francis I Proctor Foundation, University of California, San Francisco, San Francisco, CA, USA.
¹⁶ Department of Medicine, Harvard Medical School, Boston, MA, USA.
¹⁷ Division of Infectious Diseases, Massachusetts General Hospital, Boston, MA, USA.
¹⁸ Joint senior authors.
¹⁹ Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, MI 48109, USA wiensj@umich.edu.

PMID: 35177406
PMCID: PMC8850910
DOI: 10.1136/bmj-2021-068576

Multicenter Study

Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study

Fahad Kamran et al. BMJ. 2022.

. 2022 Feb 17:376:e068576.

doi: 10.1136/bmj-2021-068576.

Authors

Affiliations

¹ Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, MI 48109, USA.
² Joint first authors.
³ Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor, MI, USA.
⁴ Medical Scientist Training Program, University of Michigan Medical School, Ann Arbor, MI, USA.
⁵ Mass General Brigham Digital Health eCare, Somerville, MA, USA.
⁶ Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX, USA.
⁷ Clinical Informatics Center, University of Texas Southwestern Medical Center, Dallas, TX, USA.
⁸ Center for Clinical Informatics and Improvement Research, University of California, San Francisco, CA, USA.
⁹ Department of Emergency Medicine, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA.
¹⁰ Division of Hospital Medicine, University of California, San Francisco, San Francisco, CA, USA.
¹¹ Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor, MI, USA.
¹² Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, MI, USA.
¹³ Infection Control Unit, Massachusetts General Hospital, Boston, MA, USA.
¹⁴ Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA.
¹⁵ Francis I Proctor Foundation, University of California, San Francisco, San Francisco, CA, USA.
¹⁶ Department of Medicine, Harvard Medical School, Boston, MA, USA.
¹⁷ Division of Infectious Diseases, Massachusetts General Hospital, Boston, MA, USA.
¹⁸ Joint senior authors.
¹⁹ Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, MI 48109, USA wiensj@umich.edu.

PMID: 35177406
PMCID: PMC8850910
DOI: 10.1136/bmj-2021-068576

Abstract

Objective: To create and validate a simple and transferable machine learning model from electronic health record data to accurately predict clinical deterioration in patients with covid-19 across institutions, through use of a novel paradigm for model development and code sharing.

Design: Retrospective cohort study.

Setting: One US hospital during 2015-21 was used for model training and internal validation. External validation was conducted on patients admitted to hospital with covid-19 at 12 other US medical centers during 2020-21.

Participants: 33 119 adults (≥18 years) admitted to hospital with respiratory distress or covid-19.

Main outcome measures: An ensemble of linear models was trained on the development cohort to predict a composite outcome of clinical deterioration within the first five days of hospital admission, defined as in-hospital mortality or any of three treatments indicating severe illness: mechanical ventilation, heated high flow nasal cannula, or intravenous vasopressors. The model was based on nine clinical and personal characteristic variables selected from 2686 variables available in the electronic health record. Internal and external validation performance was measured using the area under the receiver operating characteristic curve (AUROC) and the expected calibration error-the difference between predicted risk and actual risk. Potential bed day savings were estimated by calculating how many bed days hospitals could save per patient if low risk patients identified by the model were discharged early.

Results: 9291 covid-19 related hospital admissions at 13 medical centers were used for model validation, of which 1510 (16.3%) were related to the primary outcome. When the model was applied to the internal validation cohort, it achieved an AUROC of 0.80 (95% confidence interval 0.77 to 0.84) and an expected calibration error of 0.01 (95% confidence interval 0.00 to 0.02). Performance was consistent when validated in the 12 external medical centers (AUROC range 0.77-0.84), across subgroups of sex, age, race, and ethnicity (AUROC range 0.78-0.84), and across quarters (AUROC range 0.73-0.83). Using the model to triage low risk patients could potentially save up to 7.8 bed days per patient resulting from early discharge.

Conclusion: A model to predict clinical deterioration was developed rapidly in response to the covid-19 pandemic at a single hospital, was applied externally without the sharing of data, and performed well across multiple medical centers, patient subgroups, and time periods, showing its potential as a tool for use in optimizing healthcare resources.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from National Science Foundation (NSF), National Institutes of Health (NIH) -National Library of Medicine (NLM) and -National Heart, Lung, and Blood Institute (NHLBI), Agency for Healthcare Research and Quality (AHRQ), Centers for Disease Control and Prevention (CDC) -National Center for Emerging and Zoonotic Infectious Diseases (NCEZID), Precision Health at the University of Michigan, and the Institute for Healthcare Policy and Innovation at the University of Michigan. JZA received grant funding from National Institute on Aging, Michigan Department of Health and Human Services, and Merck Foundation, outside of the submitted work; JZA also received personal fees for consulting at JAMA Network and New England Journal of Medicine, honorariums from Harvard University, University of Chicago, and University of California San Diego, and monetary support for travel reimbursements from NIH, National Academy of Medicine, and AcademyHealth, during the conduct of the study; JZA also served as a board member of AcademyHealth, Physicians Health Plan, and Center for Health Research and Transformation, with no compensation, during the conduct of the study. SB reports receiving grant funding from NIH, outside of the submitted work. JPD reports receiving personal fees from the Annals of Emergency Medicine, during the conduct of the study. RJM reports receiving grant funding from Verily Life Sciences, Sergey Brin Family Foundation, and Texas Health Resources Clinical Scholar, outside of the submitted work; RJM also served on the advisory committee of Infectious Diseases Society of America - Digital Strategy Advisory Group, during the conduct of the study. BKN reports receiving grant funding from NIH, Veterans Affairs -Health Services Research and Development Service, the American Heart Association (AHA), Janssen, and Apple, outside of the submitted work; BKN also received compensation as editor in chief of Circulation: Cardiovascular Quality and Outcomes, a journal of AHA, during the conduct of the study; BKN is also a co-inventor on US Utility Patent No US15/356 012 (US20170148158A1) entitled “Automated Analysis of Vasculature in Coronary Angiograms,” that uses software technology with signal processing and machine learning to automate the reading of coronary angiograms, held by the University of Michigan; the patent is licensed to AngioInsight, in which BKN holds ownership shares and receives consultancy fees. EÖ reports having a patent pending for the University of Michigan for an artificial intelligence based approach for the dynamic prediction of health states for patients with occupational injuries. SNS reports serving on the editorial board for the Journal of the American Medical Informatics Association, and on the student editorial board for Applied Informatics Journal, during the conduct of the study. KS reports receiving grant funding from Blue Cross Blue Shield of Michigan, and Teva Pharmaceuticals, outside of the submitted work; KS also serves on a scientific advisory board for Flatiron Health, where he receives consulting fees and honorariums for invited lectures, during the conduct of the study. MWS reports serving on the planning committee for the Machine Learning for Healthcare Conference (MLHC), a non-profit organization that hosts a yearly academic meeting. JW reports receiving grant funding from Cisco Systems, D Dan and Betty Kahn Foundation, and Alfred P Sloan Foundation, during the conduct of the study outside of the submitted work; JW also served on the international advisory board for Lancet Digital Health, and on the advisory board for MLHC, during the conduct of the study. No other disclosures were reported that could appear to have influenced the submitted work. SD, JG, FK, BYL, XL, DSM, ESS, ST, TSV, and LRW all declare: no additional support from any organization for the submitted work; no additional financial relationships with any organizations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.

Figures

**Fig 1**
Model performance across internal and external validation cohorts. Discriminative performance was measured using receiver operating characteristic curves and precision-recall curves. Model calibration is shown in reliability plots based on quintiles of predicted scores. The table summarizes results with 95% confidence intervals. The thick line shows the internal validation cohort at Michigan Medicine (MM) and the different colors represent the external validation cohorts (A-G). PPV=positive predictive value; AUROC=area under the receiver operating characteristics curve; AUPR=area under the precision-recall curve; ECE=expected calibration error

**Fig 2**
Model discriminative performance (area under the receiver operating characteristics curve (AUROC) and area under the precision-recall curve (AUPR) scores) over the year (March 2020 to February 2021) by quarter. The table shows the number (percentage) of patient hospital admissions in each cohort in each quarter and met the primary outcome of a composite of clinical deterioration within the first five days of hospital admission, defined as in-hospital mortality or any of three treatments indicating severe illness: mechanical ventilation, heated high flow nasal cannula, and intravenous vasopressors. MM=Michigan Medicine; A-G represent the external validation cohorts

**Fig 3**
Model discriminative performance (area under the receiver operating characteristics curve (AUROC) scores) evaluated across subgroups. Values are macro-average performance across institutions (error bars are ±1 standard deviation). No error bar shown for age subgroup 18-25 years because only a single institution had enough positive cases to calculate the AUROC score

**Fig 4**
Model used to identify potential patients with covid-19 for early discharge after 48 hours of observation. A decision threshold was chosen that achieves a negative predictive value of ≥95%. Figure depicts both the proportion of patients who could be discharged early and the number of bed days saved, normalized by the number of correctly discharged patients in each validation cohort. Results are computed over 1000 bootstrap replications. MM=Michigan Medicine; A-G represent the external validation cohorts

See this image and copyright information in PMC

Comment in

Predicting covid-19 outcomes.
Habib AR, Lo NC. Habib AR, et al. BMJ. 2022 Feb 17;376:o354. doi: 10.1136/bmj.o354. BMJ. 2022. PMID: 35177413 No abstract available.

Cited by

Defining medical liability when artificial intelligence is applied on diagnostic algorithms: a systematic review.
Cestonaro C, Delicati A, Marcante B, Caenazzo L, Tozzo P. Cestonaro C, et al. Front Med (Lausanne). 2023 Nov 27;10:1305756. doi: 10.3389/fmed.2023.1305756. eCollection 2023. Front Med (Lausanne). 2023. PMID: 38089864 Free PMC article. Review.
Predicting Severe Respiratory Failure in Patients with COVID-19: A Machine Learning Approach.
Ceylan B, Olmuşçelik O, Karaalioğlu B, Ceylan Ş, Şahin M, Aydın S, Yılmaz E, Dumlu R, Kapmaz M, Çiçek Y, Kansu A, Duger M, Mert A. Ceylan B, et al. J Clin Med. 2024 Dec 4;13(23):7386. doi: 10.3390/jcm13237386. J Clin Med. 2024. PMID: 39685844 Free PMC article.
From Biased Selective Labels to Pseudo-Labels: An Expectation-Maximization Framework for Learning from Biased Decisions.
Chang T, Wiens J. Chang T, et al. Proc Mach Learn Res. 2024 Jul;235:6286-6324. Proc Mach Learn Res. 2024. PMID: 40574796 Free PMC article.
Transforming Cardiovascular Care With Artificial Intelligence: From Discovery to Practice: JACC State-of-the-Art Review.
Khera R, Oikonomou EK, Nadkarni GN, Morley JR, Wiens J, Butte AJ, Topol EJ. Khera R, et al. J Am Coll Cardiol. 2024 Jul 2;84(1):97-114. doi: 10.1016/j.jacc.2024.05.003. J Am Coll Cardiol. 2024. PMID: 38925729 Free PMC article. Review.
Methodological challenges using routine clinical care data for real-world evidence: a rapid review utilizing a systematic literature search and focus group discussion.
Pfaffenlehner M, Behrens M, Zöller D, Ungethüm K, Günther K, Rücker V, Reese JP, Heuschmann P, Kesselmeier M, Remo F, Scherag A, Binder H, Binder N; EVA4MII project. Pfaffenlehner M, et al. BMC Med Res Methodol. 2025 Jan 14;25(1):8. doi: 10.1186/s12874-024-02440-x. BMC Med Res Methodol. 2025. PMID: 39810151 Free PMC article.

See all "Cited by" articles

References

1. Shah NH, Milstein A, Bagley PhD SC. Making Machine Learning Models Clinically Useful. JAMA 2019;322:1351-2. 10.1001/jama.2019.10306. - DOI - PubMed
1. Peterson ED. Machine Learning, Predictive Analytics, and Clinical Practice: Can the Past Inform the Present? JAMA 2019;322:2283-4. 10.1001/jama.2019.17831. - DOI - PubMed
1. White DB, Lo B. A Framework for Rationing Ventilators and Critical Care Beds During the COVID-19 Pandemic. JAMA 2020;323:1773-4. 10.1001/jama.2020.5046. - DOI - PubMed
1. Coley CM, Li Y-H, Medsger AR, et al. . Preferences for home vs hospital care among low-risk patients with community-acquired pneumonia. Arch Intern Med 1996;156:1565-71. 10.1001/archinte.1996.00440130115012. - DOI - PubMed
1. Page K, Barnett AG, Graves N. What is a hospital bed day worth? A contingent valuation study of hospital Chief Executive Officers. BMC Health Serv Res 2017;17:137. 10.1186/s12913-017-2079-5. - DOI - PMC - PubMed

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Medical
- MedlinePlus Health Information

[1] Shah NH, Milstein A, Bagley PhD SC. Making Machine Learning Models Clinically Useful. JAMA 2019;322:1351-2. 10.1001/jama.2019.10306. - DOI - PubMed

[2] Shah NH, Milstein A, Bagley PhD SC. Making Machine Learning Models Clinically Useful. JAMA 2019;322:1351-2. 10.1001/jama.2019.10306. - DOI - PubMed

[3] Peterson ED. Machine Learning, Predictive Analytics, and Clinical Practice: Can the Past Inform the Present? JAMA 2019;322:2283-4. 10.1001/jama.2019.17831. - DOI - PubMed

[4] Peterson ED. Machine Learning, Predictive Analytics, and Clinical Practice: Can the Past Inform the Present? JAMA 2019;322:2283-4. 10.1001/jama.2019.17831. - DOI - PubMed

[5] White DB, Lo B. A Framework for Rationing Ventilators and Critical Care Beds During the COVID-19 Pandemic. JAMA 2020;323:1773-4. 10.1001/jama.2020.5046. - DOI - PubMed

[6] White DB, Lo B. A Framework for Rationing Ventilators and Critical Care Beds During the COVID-19 Pandemic. JAMA 2020;323:1773-4. 10.1001/jama.2020.5046. - DOI - PubMed

[7] Coley CM, Li Y-H, Medsger AR, et al. . Preferences for home vs hospital care among low-risk patients with community-acquired pneumonia. Arch Intern Med 1996;156:1565-71. 10.1001/archinte.1996.00440130115012. - DOI - PubMed

[8] Coley CM, Li Y-H, Medsger AR, et al. . Preferences for home vs hospital care among low-risk patients with community-acquired pneumonia. Arch Intern Med 1996;156:1565-71. 10.1001/archinte.1996.00440130115012. - DOI - PubMed

[9] Page K, Barnett AG, Graves N. What is a hospital bed day worth? A contingent valuation study of hospital Chief Executive Officers. BMC Health Serv Res 2017;17:137. 10.1186/s12913-017-2079-5. - DOI - PMC - PubMed

[10] Page K, Barnett AG, Graves N. What is a hospital bed day worth? A contingent valuation study of hospital Chief Executive Officers. BMC Health Serv Res 2017;17:137. 10.1186/s12913-017-2079-5. - DOI - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study

Affiliations

Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Comment in

Similar articles

Cited by

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Medical