Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Dec 12;19(12):e0312968.
doi: 10.1371/journal.pone.0312968. eCollection 2024.

Systematic evaluation of machine learning models for postoperative surgical site infection prediction

Anna M van Boekel  1 , Siri L van der Meijden  2   3 , Sesmu M Arbous  2 , Rob G H H Nelissen  4 , Karin E Veldkamp  5 , Emma B Nieswaag  2   3 , Kim F T Jochems  2   3 , Jeroen Holtz  2   3 , Annekee van IJlzinga Veenstra  2   3 , Jeroen Reijman  2   3 , Ype de Jong  1   6 , Harry van Goor  7 , Maryse A Wiewel  3 , Jan W Schoones  8 , Bart F Geerts  3 , Mark G J de Boer  6   9
Affiliations

Systematic evaluation of machine learning models for postoperative surgical site infection prediction

Anna M van Boekel et al. PLoS One. .

Abstract

Background: Surgical site infections (SSIs) lead to increased mortality and morbidity, as well as increased healthcare costs. Multiple models for the prediction of this serious surgical complication have been developed, with an increasing use of machine learning (ML) tools.

Objective: The aim of this systematic review was to assess the performance as well as the methodological quality of validated ML models for the prediction of SSIs.

Methods: A systematic search in PubMed, Embase and the Cochrane library was performed from inception until July 2023. Exclusion criteria were the absence of reported model validation, SSIs as part of a composite adverse outcome, and pediatric populations. ML performance measures were evaluated, and ML performances were compared to regression-based methods for studies that reported both methods. Risk of bias (ROB) of the studies was assessed using the Prediction model Risk of Bias Assessment Tool.

Results: Of the 4,377 studies screened, 24 were included in this review, describing 85 ML models. Most models were only internally validated (81%). The C-statistic was the most used performance measure (reported in 96% of the studies) and only two studies reported calibration metrics. A total of 116 different predictors were described, of which age, steroid use, sex, diabetes, and smoking were most frequently (100% to 75%) incorporated. Thirteen studies compared ML models to regression-based models and showed a similar performance of both modelling methods. For all included studies, the overall ROB was high or unclear.

Conclusions: A multitude of ML models for the prediction of SSIs are available, with large variability in performance. However, most models lacked external validation, performance was reported limitedly, and the risk of bias was high. In studies describing both ML models and regression-based models, one modelling method did not outperform the other.

PubMed Disclaimer

Conflict of interest statement

I have read the journal’s policy and the authors of this manuscript have the following competing interests: B.F. Geerts declares to be shareholder and owner of Healthplus.ai S.L. van der Meijden, M. Wiewel, E.B. Nieswaag, K.F.T. Jochems, J. Holtz, A. van IJlzinga Veenstra, and J. Reijman declare to be an employee of Healthplus.ai. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

Figures

Fig 1
Fig 1. PRISMA figure.
Fig 2
Fig 2. Predictors used in proportion of the ML models.
All predictors used five times or more are included in the figure. ASA classification (American Society of Anesthesiologists); BMI, (Body Mass Index); COPD (Chronic Obstructive Pulmonary Disease); INR, (International Normalized Ratio); PT, (Prothrombin time); WBC, (White blood count).
Fig 3
Fig 3. Area under the curve (AUC) for each article that presented both ML and regression-based models.
Green dots represent the AUC of the ML models, orange dots represent the AUC of the regression-based models. The green and orange lines represent the median.
Fig 4
Fig 4. Summary of risk of bias assessment using the PROBAST.
Green low risk of bias, yellow unclear risk of bias due to lack of information, red high risk of bias. ROB; Risk of bias.
See this image and copyright information in PMC

References

    1. Global patient outcomes after elective surgery: prospective cohort study in 27 low-, middle- and high-income countries. Br J Anaesth. 2016;117(5):601–9. - PMC - PubMed
    1. Badia JM, Casey AL, Petrosillo N, Hudson PM, Mitchell SA, Crosby C. Impact of surgical site infection on healthcare costs and patient outcomes: a systematic review in six European countries. J Hosp Infect. 2017;96(1):1–15. doi: 10.1016/j.jhin.2017.03.004 - DOI - PubMed
    1. Gillespie BM, Harbeck E, Rattray M, Liang R, Walker R, Latimer S, et al.. Worldwide incidence of surgical site infections in general surgical patients: A systematic review and meta-analysis of 488,594 patients. Int J Surg. 2021;95:106136. doi: 10.1016/j.ijsu.2021.106136 - DOI - PubMed
    1. European Centre for Disease Prevention and Control. Healthcare-associated infections: surgical site infections. ECDC. Annual epidemiological report for 2018–2020. Stockholm; 2023.
    1. Qu H, Liu Y, Bi DS. Clinical risk factors for anastomotic leakage after laparoscopic anterior resection for rectal cancer: a systematic review and meta-analysis. Surg Endosc. 2015;29(12):3608–17. doi: 10.1007/s00464-015-4117-x - DOI - PubMed

Publication types

LinkOut - more resources