Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records

Iacopo Vagliano¹, Martijn C Schut², Ameen Abu-Hanna², Dave A Dongelmans³, Dylan W de Lange⁴, Diederik Gommers⁵, Olaf L Cremer⁶, Rob J Bosman⁷, Sander Rigter⁸, Evert-Jan Wils⁹, Tim Frenzel¹⁰, Remko de Jong¹¹, Marco A A Peters¹², Marlijn J A Kamps¹³, Dharmanand Ramnarain¹⁴, Ralph Nowitzky¹⁵, Fleur G C A Nooteboom¹⁶, Wouter de Ruijter¹⁷, Louise C Urlings-Strop¹⁸, Ellen G M Smit¹⁹, D Jannet Mehagnoul-Schipper²⁰, Tom Dormans²¹, Cornelis P C de Jager²², Stefaan H A Hendriks²³, Sefanja Achterberg²⁴, Evelien Oostdijk²⁵, Auke C Reidinga²⁶, Barbara Festen-Spanjer²⁷, Gert B Brunnekreef²⁸, Alexander D Cornet²⁹, Walter van den Tempel³⁰, Age D Boelens³¹, Peter Koetsier³², Judith Lens³³, Harald J Faber³⁴, A Karakus³⁵, Robert Entjes³⁶, Paul de Jong³⁷, Thijs C D Rettig³⁸, M C Reuland², Sesmu Arbous³⁹, Lucas M Fleuren⁴⁰, Tariq A Dam⁴⁰, Patrick J Thoral⁴⁰, Robbert C A Lalisang⁴¹, Michele Tonutti⁴¹, Daan P de Bruin⁴¹, Paul W G Elbers⁴⁰, Nicolette F de Keizer⁴²; Dutch COVID-19 Research Consortium, the Dutch ICU Data Sharing Against COVID-19 Collaborators

Affiliations

¹ Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands. Electronic address: i.vagliano@amsterdamumc.nl.
² Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands.
³ National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands.
⁴ National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands.
⁵ Department of Intensive Care, Erasmus Medical Center, Rotterdam, The Netherlands.
⁶ Intensive Care, UMC Utrecht, Utrecht, The Netherlands.
⁷ ICU, OLVG, Amsterdam, The Netherlands.
⁸ Department of Anesthesiology and Intensive Care, St. Antonius Hospital, Nieuwegein, The Netherlands.
⁹ Department of Intensive Care, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands.
¹⁰ Department of Intensive Care Medicine, Radboud University Medical Center, Nijmegen, The Netherlands.
¹¹ Intensive Care, Bovenij Ziekenhuis, Amsterdam, The Netherlands.
¹² Intensive Care, Canisius Wilhelmina Ziekenhuis, Nijmegen, The Netherlands.
¹³ Intensive Care, Catharina Ziekenhuis Eindhoven, Eindhoven, The Netherlands.
¹⁴ Department of Intensive Care, ETZ Tilburg, Tilburg, The Netherlands.
¹⁵ Intensive Care, Haga Ziekenhuis, Den Haag, The Netherlands.
¹⁶ Intensive Care, Laurentius Ziekenhuis, Roermond, The Netherlands.
¹⁷ Department of Intensive Care Medicine, Northwest Clinics, Alkmaar, The Netherlands.
¹⁸ Intensive Care, Reinier de Graaf Gasthuis, Delft, The Netherlands.
¹⁹ Intensive Care, Spaarne Gasthuis, Haarlem en Hoofddorp, The Netherlands.
²⁰ Intensive Care, VieCuri Medisch Centrum, Venlo, The Netherlands.
²¹ Intensive care, Zuyderland MC, Heerlen, The Netherlands.
²² Department of Intensive Care, Jeroen Bosch Ziekenhuis, Den Bosch, The Netherlands.
²³ Intensive Care, Albert Schweitzerziekenhuis, Dordrecht, The Netherlands.
²⁴ ICU, Haaglanden Medisch Centrum, Den Haag, The Netherlands.
²⁵ ICU, Maasstad Ziekenhuis Rotterdam, Rotterdam, The Netherlands.
²⁶ ICU, SEH, BWC, Martiniziekenhuis, Groningen, The Netherlands.
²⁷ Intensive Care, Ziekenhuis Gelderse Vallei, Ede, The Netherlands.
²⁸ Department of Intensive Care, Ziekenhuisgroep Twente, Almelo, The Netherlands.
²⁹ Department of Intensive Care, Medisch Spectrum Twente, Enschede, The Netherlands.
³⁰ Department of Intensive Care, Ikazia Ziekenhuis Rotterdam, Rotterdam, The Netherlands.
³¹ Anesthesiology, Antonius Ziekenhuis Sneek, Sneek, The Netherlands.
³² Intensive Care, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands.
³³ ICU, IJsselland Ziekenhuis, Capelle aan den IJssel, The Netherlands.
³⁴ ICU, WZA, Assen, The Netherlands.
³⁵ Department of Intensive Care, Diakonessenhuis Hospital, Utrecht, The Netherlands.
³⁶ Department of Intensive Care, Adrz, Goes, The Netherlands.
³⁷ Department of Anesthesia and Intensive Care, Slingeland Ziekenhuis, Doetinchem, The Netherlands.
³⁸ Department of Anesthesiology, Intensive Care and Pain Medicine, Amphia Ziekenhuis, Breda, The Netherlands.
³⁹ Intensivist, LUMC, Leiden, The Netherlands.
⁴⁰ Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands.
⁴¹ Pacmed, Amsterdam, The Netherlands.
⁴² Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands; National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands.

PMID: 36162166
PMCID: PMC9492397
DOI: 10.1016/j.ijmedinf.2022.104863

Observational Study

Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records

Iacopo Vagliano et al. Int J Med Inform. 2022 Nov.

. 2022 Nov:167:104863.

doi: 10.1016/j.ijmedinf.2022.104863. Epub 2022 Sep 22.

Authors

Affiliations

¹ Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands. Electronic address: i.vagliano@amsterdamumc.nl.
² Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands.
³ National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands.
⁴ National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands.
⁵ Department of Intensive Care, Erasmus Medical Center, Rotterdam, The Netherlands.
⁶ Intensive Care, UMC Utrecht, Utrecht, The Netherlands.
⁷ ICU, OLVG, Amsterdam, The Netherlands.
⁸ Department of Anesthesiology and Intensive Care, St. Antonius Hospital, Nieuwegein, The Netherlands.
⁹ Department of Intensive Care, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands.
¹⁰ Department of Intensive Care Medicine, Radboud University Medical Center, Nijmegen, The Netherlands.
¹¹ Intensive Care, Bovenij Ziekenhuis, Amsterdam, The Netherlands.
¹² Intensive Care, Canisius Wilhelmina Ziekenhuis, Nijmegen, The Netherlands.
¹³ Intensive Care, Catharina Ziekenhuis Eindhoven, Eindhoven, The Netherlands.
¹⁴ Department of Intensive Care, ETZ Tilburg, Tilburg, The Netherlands.
¹⁵ Intensive Care, Haga Ziekenhuis, Den Haag, The Netherlands.
¹⁶ Intensive Care, Laurentius Ziekenhuis, Roermond, The Netherlands.
¹⁷ Department of Intensive Care Medicine, Northwest Clinics, Alkmaar, The Netherlands.
¹⁸ Intensive Care, Reinier de Graaf Gasthuis, Delft, The Netherlands.
¹⁹ Intensive Care, Spaarne Gasthuis, Haarlem en Hoofddorp, The Netherlands.
²⁰ Intensive Care, VieCuri Medisch Centrum, Venlo, The Netherlands.
²¹ Intensive care, Zuyderland MC, Heerlen, The Netherlands.
²² Department of Intensive Care, Jeroen Bosch Ziekenhuis, Den Bosch, The Netherlands.
²³ Intensive Care, Albert Schweitzerziekenhuis, Dordrecht, The Netherlands.
²⁴ ICU, Haaglanden Medisch Centrum, Den Haag, The Netherlands.
²⁵ ICU, Maasstad Ziekenhuis Rotterdam, Rotterdam, The Netherlands.
²⁶ ICU, SEH, BWC, Martiniziekenhuis, Groningen, The Netherlands.
²⁷ Intensive Care, Ziekenhuis Gelderse Vallei, Ede, The Netherlands.
²⁸ Department of Intensive Care, Ziekenhuisgroep Twente, Almelo, The Netherlands.
²⁹ Department of Intensive Care, Medisch Spectrum Twente, Enschede, The Netherlands.
³⁰ Department of Intensive Care, Ikazia Ziekenhuis Rotterdam, Rotterdam, The Netherlands.
³¹ Anesthesiology, Antonius Ziekenhuis Sneek, Sneek, The Netherlands.
³² Intensive Care, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands.
³³ ICU, IJsselland Ziekenhuis, Capelle aan den IJssel, The Netherlands.
³⁴ ICU, WZA, Assen, The Netherlands.
³⁵ Department of Intensive Care, Diakonessenhuis Hospital, Utrecht, The Netherlands.
³⁶ Department of Intensive Care, Adrz, Goes, The Netherlands.
³⁷ Department of Anesthesia and Intensive Care, Slingeland Ziekenhuis, Doetinchem, The Netherlands.
³⁸ Department of Anesthesiology, Intensive Care and Pain Medicine, Amphia Ziekenhuis, Breda, The Netherlands.
³⁹ Intensivist, LUMC, Leiden, The Netherlands.
⁴⁰ Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands.
⁴¹ Pacmed, Amsterdam, The Netherlands.
⁴² Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands; National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands.

PMID: 36162166
PMCID: PMC9492397
DOI: 10.1016/j.ijmedinf.2022.104863

Abstract

Purpose: To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data.

Methods: Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors.

Results: A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors.

Conclusion: In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.

Keywords: Covid-19 [C01.748.610.763.500]; Critical care [E02.760.190]; Electronic Health Record [E05.318.308.940.968.625.500]; In-hospital mortality [E05.318.308.985.550.400]; Machine learning [G17.035.250.500]; Prognosis [E01.789].

PubMed Disclaimer

Conflict of interest statement

Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Figures

**Fig. 1**
Calibration curves of the REG and EHR models in the internal as well as temporal validations.

**Fig. 2**
Coefficients of the EHR logistic regression model. PaCO₂ is the partial pressure of carbon dioxide, FiO₂ is the fraction of inspired oxygen. Supplementary Table S2 includes the model description.

**Fig. 3**
Coefficients of the REG logistic regression model. Supplementary Table S3 includes the model description.

See this image and copyright information in PMC

References

1. L.B. Wynants, G.S. Van Calster, R.D. Collins, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal, BMJ, 369: m1328. - PMC - PubMed
1. Grasselli G., Greco M., Zanella A., et al. Risk Factors Associated With Mortality Among Patients With COVID-19 in Intensive Care Units in Lombardy. Italy. JAMA Intern Med. 2020 doi: 10.1001/jamainternmed.2020.3539. - DOI - PMC - PubMed
1. Richardson S., Hirsch J.S., Narasimhan M., et al. Presenting Characteristics, Comorbidities, and Outcomes Among 5700 Patients Hospitalized With COVID-19 in the New York City Area. JAMA. 2020;323:2052–2059. doi: 10.1001/jama.2020.6775. - DOI - PMC - PubMed
1. Karagiannidis C., Mostert C., Hentschker C., et al. Case characteristics, resource use, and outcomes of 10 021 patients with COVID-19 admitted to 920 German hospitals: an observational study. Lancet Respir Med. 2020;8:853–862. doi: 10.1016/S2213-2600(20)30316-7. - DOI - PMC - PubMed
1. Fleuren L.M., de Bruin D.P., Tonutti M., Lalisang R.C.A., Elbers P.W.G. Dutch ICU Data Sharing Collaborators. Large-scale ICU data sharing for global collaboration: the first 1633 critically ill COVID-19 patients in the Dutch Data Warehouse. Intensive Care Med. 2021 - PMC - PubMed

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- ClinicalKey

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records

Affiliations

Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources