Evaluating machine learning models for sepsis prediction: A systematic review of methodologies

doi:10.1016/j.isci.2021.103651

. 2021 Dec 20;25(1):103651.

doi: 10.1016/j.isci.2021.103651. eCollection 2022 Jan 21.

Evaluating machine learning models for sepsis prediction: A systematic review of methodologies

Hong-Fei Deng^{1

2}, Ming-Wei Sun³, Yu Wang^{1

2

3

4}, Jun Zeng^{1

2

3

4}, Ting Yuan^{1

2}, Ting Li^{1

2}, Di-Huan Li^{1

2}, Wei Chen⁵, Ping Zhou⁶, Qi Wang⁷, Hua Jiang^{1

2

3

4}

Affiliations

¹ Institute for Emergency and Disaster Medicine, Sichuan Academy of Medical Sciences, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan 610072, China.
² School of Medicine, University of Electronic Science and Technology of China, Chengdu 610054, China.
³ Emergency Center of Sichuan Provincial People's Hospital, Sichuan Academy of Medical Sciences, Chengdu 610072, China.
⁴ Sichuan Clinical Research Center for Emergency and Critical Care, Chengdu, Sichuan 610072, China.
⁵ Department of Clinical Nutrition, Peking Union Medical College Hospital, Beijing 100730, China.
⁶ Emergency Intensive Care Unit of Sichuan Provincial People's Hospital, Sichuan Academy of Medical Sciences, Chengdu 610072, China.
⁷ Beijing Computational Science Research Center, Beijing 100193, China.

PMID: 35028534
PMCID: PMC8741489
DOI: 10.1016/j.isci.2021.103651

Evaluating machine learning models for sepsis prediction: A systematic review of methodologies

Hong-Fei Deng et al. iScience. 2021.

. 2021 Dec 20;25(1):103651.

doi: 10.1016/j.isci.2021.103651. eCollection 2022 Jan 21.

Authors

Hong-Fei Deng^{1

2}, Ming-Wei Sun³, Yu Wang^{1

2

3

4}, Jun Zeng^{1

2

3

4}, Ting Yuan^{1

2}, Ting Li^{1

2}, Di-Huan Li^{1

2}, Wei Chen⁵, Ping Zhou⁶, Qi Wang⁷, Hua Jiang^{1

2

3

4}

Affiliations

¹ Institute for Emergency and Disaster Medicine, Sichuan Academy of Medical Sciences, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan 610072, China.
² School of Medicine, University of Electronic Science and Technology of China, Chengdu 610054, China.
³ Emergency Center of Sichuan Provincial People's Hospital, Sichuan Academy of Medical Sciences, Chengdu 610072, China.
⁴ Sichuan Clinical Research Center for Emergency and Critical Care, Chengdu, Sichuan 610072, China.
⁵ Department of Clinical Nutrition, Peking Union Medical College Hospital, Beijing 100730, China.
⁶ Emergency Intensive Care Unit of Sichuan Provincial People's Hospital, Sichuan Academy of Medical Sciences, Chengdu 610072, China.
⁷ Beijing Computational Science Research Center, Beijing 100193, China.

PMID: 35028534
PMCID: PMC8741489
DOI: 10.1016/j.isci.2021.103651

Abstract

Studies for sepsis prediction using machine learning are developing rapidly in medical science recently. In this review, we propose a set of new evaluation criteria and reporting standards to assess 21 qualified machine learning models for quality analysis based on PRISMA. Our assessment shows that (1.) the definition of sepsis is not consistent among the studies; (2.) data sources and data preprocessing methods, machine learning models, feature engineering, and inclusion types vary widely among the studies; (3.) the closer to the onset of sepsis, the higher the value of AUROC is; (4.) the improvement in AUROC is primarily due to using machine learning as a feature engineering tool; (5.) deep neural networks coupled with Sepsis-3 diagnostic criteria tend to yield better results on the time series data collected from patients with sepsis. The new evaluation criteria and reporting standards will facilitate the development of improved machine learning models for clinical applications.

Keywords: Clinical medicine; Machine learning.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

**Figure 1**
Literature screening flowchart

**Figure 2**
Predicting performance of multi-time points, related to Table 3

See this image and copyright information in PMC

Cited by

Decrease in Mortality after the Implementation of a Hospital Model to Improve Performance in Sepsis Care: Princess Sepsis Code.
Méndez R, Figuerola A, Ramasco F, Chicot M, Pascual NF, García Í, von Wernitz A, Zurita ND, Semiglia A, Pizarro A, Saez C, Rodríguez D. Méndez R, et al. J Pers Med. 2024 Jan 29;14(2):149. doi: 10.3390/jpm14020149. J Pers Med. 2024. PMID: 38392582 Free PMC article.
Machine Learning-Based Early Prediction of Sepsis Using Electronic Health Records: A Systematic Review.
Islam KR, Prithula J, Kumar J, Tan TL, Reaz MBI, Sumon MSI, Chowdhury MEH. Islam KR, et al. J Clin Med. 2023 Aug 30;12(17):5658. doi: 10.3390/jcm12175658. J Clin Med. 2023. PMID: 37685724 Free PMC article. Review.
Hospital Acquired Sepsis, Disease Prevalence, and Recent Advances in Sepsis Mitigation.
Garvey M. Garvey M. Pathogens. 2024 May 30;13(6):461. doi: 10.3390/pathogens13060461. Pathogens. 2024. PMID: 38921759 Free PMC article. Review.
Development and Validation of a Machine Learning Model for Early Detection of Untreated Infection.
Buell KG, Carey KA, Dussault N, Parker WF, Dumanian J, Bhavani SV, Gilbert ER, Winslow CJ, Shah NS, Afshar M, Edelson DP, Churpek MM. Buell KG, et al. Crit Care Explor. 2024 Oct 11;6(10):e1165. doi: 10.1097/CCE.0000000000001165. eCollection 2024 Oct 1. Crit Care Explor. 2024. PMID: 39392375 Free PMC article.
Machine learning model for daily prediction of pediatric sepsis using Phoenix criteria.
Chanci D, Grunwell JR, Rafiei A, Brown SR, Ripple MJ, Bishop NR, Rajapreyar P, Lima LM, Kamaleswaran R. Chanci D, et al. Pediatr Res. 2025 Jun 19. doi: 10.1038/s41390-025-04221-8. Online ahead of print. Pediatr Res. 2025. PMID: 40533506

See all "Cited by" articles

References

1. Alhazzani W., Møller M.H., Arabi Y.M., Loeb M., Gong M.N., Fan E., Oczkowski S., Levy M.M., Derde L., Dzierba A., et al. Surviving sepsis campaign: guidelines on the management of critically Ill adults with coronavirus disease 2019 (COVID-19) Crit. Care Med. 2020;48:e440–e469. doi: 10.1097/ccm.0000000000004363. - DOI - PMC - PubMed
1. Barton C., Chettipally U., Zhou Y., Jiang Z., Lynn-Palevsky A., Le S., Calvert J., Das R. Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Comput. Biol. Med. 2019;109:79–84. doi: 10.1016/j.compbiomed.2019.04.027. - DOI - PMC - PubMed
1. Beaulieu-Jones B.K., Lavage D.R., Snyder J.W., Moore J.H., Pendergrass S.A., Bauer C.R. Characterizing and managing missing structured data in electronic health records: data analysis. JMIR Med. Inform. 2018;6:e11. doi: 10.2196/medinform.8960. - DOI - PMC - PubMed
1. Bedoya A.D., Futoma J., Clement M.E., Corey K., Brajer N., Lin A., Simons M.G., Gao M., Nichols M., Balu S., et al. Machine learning for early detection of sepsis: an internal and temporal validation study. JAMIAOpen. 2020;3:252–260. doi: 10.1093/jamiaopen/ooaa006. - DOI - PMC - PubMed
1. Bloch E., Rotem T., Cohen J., Singer P., Aperstein Y. Machine learning models for analysis of vital signs dynamics: a case for sepsis onset prediction. J. Healthc. Eng. 2019;2019:5930379. doi: 10.1155/2019/5930379. - DOI - PMC - PubMed

LinkOut - more resources

Full Text Sources

[1] Alhazzani W., Møller M.H., Arabi Y.M., Loeb M., Gong M.N., Fan E., Oczkowski S., Levy M.M., Derde L., Dzierba A., et al. Surviving sepsis campaign: guidelines on the management of critically Ill adults with coronavirus disease 2019 (COVID-19) Crit. Care Med. 2020;48:e440–e469. doi: 10.1097/ccm.0000000000004363. - DOI - PMC - PubMed

[2] Alhazzani W., Møller M.H., Arabi Y.M., Loeb M., Gong M.N., Fan E., Oczkowski S., Levy M.M., Derde L., Dzierba A., et al. Surviving sepsis campaign: guidelines on the management of critically Ill adults with coronavirus disease 2019 (COVID-19) Crit. Care Med. 2020;48:e440–e469. doi: 10.1097/ccm.0000000000004363. - DOI - PMC - PubMed

[3] Barton C., Chettipally U., Zhou Y., Jiang Z., Lynn-Palevsky A., Le S., Calvert J., Das R. Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Comput. Biol. Med. 2019;109:79–84. doi: 10.1016/j.compbiomed.2019.04.027. - DOI - PMC - PubMed

[4] Barton C., Chettipally U., Zhou Y., Jiang Z., Lynn-Palevsky A., Le S., Calvert J., Das R. Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Comput. Biol. Med. 2019;109:79–84. doi: 10.1016/j.compbiomed.2019.04.027. - DOI - PMC - PubMed

[5] Beaulieu-Jones B.K., Lavage D.R., Snyder J.W., Moore J.H., Pendergrass S.A., Bauer C.R. Characterizing and managing missing structured data in electronic health records: data analysis. JMIR Med. Inform. 2018;6:e11. doi: 10.2196/medinform.8960. - DOI - PMC - PubMed

[6] Beaulieu-Jones B.K., Lavage D.R., Snyder J.W., Moore J.H., Pendergrass S.A., Bauer C.R. Characterizing and managing missing structured data in electronic health records: data analysis. JMIR Med. Inform. 2018;6:e11. doi: 10.2196/medinform.8960. - DOI - PMC - PubMed

[7] Bedoya A.D., Futoma J., Clement M.E., Corey K., Brajer N., Lin A., Simons M.G., Gao M., Nichols M., Balu S., et al. Machine learning for early detection of sepsis: an internal and temporal validation study. JAMIAOpen. 2020;3:252–260. doi: 10.1093/jamiaopen/ooaa006. - DOI - PMC - PubMed

[8] Bedoya A.D., Futoma J., Clement M.E., Corey K., Brajer N., Lin A., Simons M.G., Gao M., Nichols M., Balu S., et al. Machine learning for early detection of sepsis: an internal and temporal validation study. JAMIAOpen. 2020;3:252–260. doi: 10.1093/jamiaopen/ooaa006. - DOI - PMC - PubMed

[9] Bloch E., Rotem T., Cohen J., Singer P., Aperstein Y. Machine learning models for analysis of vital signs dynamics: a case for sepsis onset prediction. J. Healthc. Eng. 2019;2019:5930379. doi: 10.1155/2019/5930379. - DOI - PMC - PubMed

[10] Bloch E., Rotem T., Cohen J., Singer P., Aperstein Y. Machine learning models for analysis of vital signs dynamics: a case for sepsis onset prediction. J. Healthc. Eng. 2019;2019:5930379. doi: 10.1155/2019/5930379. - DOI - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Evaluating machine learning models for sepsis prediction: A systematic review of methodologies

Affiliations

Evaluating machine learning models for sepsis prediction: A systematic review of methodologies

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

LinkOut - more resources

Full Text Sources

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Related information

LinkOut - more resources

Full Text Sources