Development and validation of artificial intelligence models for early detection of postoperative infections (PERISCOPE): a multicentre study using electronic health record data

Abstract

Background: Postoperative infections significantly impact patient outcomes and costs, exacerbated by late diagnoses, yet early reliable predictors are scarce. Existing artificial intelligence (AI) models for postoperative infection prediction often lack external validation or perform poorly in local settings when validated. We aimed to develop locally valid models as part of the PERISCOPE AI system to enable early detection, safer discharge, and more timely treatment of patients.

Methods: We developed and validated XGBoost models to predict postoperative infections within 7 and 30 days of surgery. Using retrospective pre-operative and intra-operative electronic health record data from 2014 to 2023 across various surgical specialities, the models were developed at Hospital A and validated and updated at Hospitals B and C in the Netherlands and Belgium. Model performance was evaluated before and after updating using the two most recent years of data as temporal validation datasets. Main outcome measures were model discrimination (area under the receiver operating characteristic curve (AUROC)), calibration (slope, intercept, and plots), and clinical utility (decision curve analysis with net benefit).

Findings: The study included 253,010 surgical procedures with 23,903 infections within 30-days. Discriminative performance, calibration properties, and clinical utility significantly improved after updating. Final AUROCs after updating for Hospitals A, B, and C were 0.82 (95% confidence interval (CI) 0.81-0.83), 0.82 (95% CI 0.81-0.83), and 0.91 (95% CI 0.90-0.91) respectively for 30-day predictions on the temporal validation datasets (2022-2023). Calibration plots demonstrated adequate correspondence between observed outcomes and predicted risk. All local models were deemed clinically useful as the net benefit was higher than default strategies (treat all and treat none) over a wide range of clinically relevant decision thresholds.

Interpretation: PERISCOPE can accurately predict overall postoperative infections within 7- and 30-days post-surgery. The robust performance implies potential for improving clinical care in diverse clinical target populations. This study supports the need for approaches to local updating of AI models to account for domain shifts in patient populations and data distributions across different clinical settings.

Funding: This study was funded by a REACT EU grant from European Regional Development Fund (ERDF) and Kansen voor West.

Keywords: Artificial intelligence; Clinical utility; Model updating; Multi-centre validation; Postoperative infection.

Fig. 1

Calibration plots of final, locally updated models on the validation datasets (years 2022–2023): a) Hospital A – 7 day predictions, b) Hospital A – 30 day predictions, c) Hospital B – 7 day predictions, d) Hospital B – 30 day predictions, e) Hospital C – 7 day predictions, and f) Hospital C – 30 day predictions. The calibration curves (upper plots) show the agreement between the predicted probabilities by the model and the outcome of patients for those predictions. The histograms in the lower plot show the distributions of predictions.

Fig. 2

Decision curves showing the net benefit for PERISCOPE's models before and after updating compared to ‘treat all’ and ‘treat none’ patients for validation datasets (years 2022–2023): a) Hospital A – 7 day predictions, b) Hospital A – 30 day predictions, c) Hospital B – 7 day predictions, d) Hospital B – 30 day predictions, e) Hospital C – 7 day predictions, and f) Hospital C – 30 day predictions. The unit of net benefit is ‘true positives’. At lower decision thresholds, the end-user is more worried about the disease, i.e., willing to accept more false positives, and at higher decision thresholds, the end-user is more worried about the intervention following the prediction. To be of clinical value, net benefit should be higher than zero and default strategies in the established range of clinically relevant risk thresholds.

Fig. 3

Model explainability in terms of SHAP (SHapley Additive exPlanations) values for: a) Hospital A – 30 day predictions, b) Hospital B – 30 day predictions, and c) Hospital C – 30 day predictions. For Hospitals B and C, updated models were used for predictions. In the SHAP plot, red dots represent high feature values and blue dots represent low feature values. Dots on the right side of the y-axis indicate a contribution to a higher predicted risk of infection, while dots on the left side indicate a lower risk. Grey dots signify missing feature values.