Incorporating preoperative frailty to assist in early prediction of postoperative pneumonia in elderly patients with hip fractures: an externally validated online interpretable machine learning model

Abstract

Background: This study aims to implement a validated prediction model and application medium for postoperative pneumonia (POP) in elderly patients with hip fractures in order to facilitate individualized intervention by clinicians.

Methods: Employing clinical data from elderly patients with hip fractures, we derived and externally validated machine learning models for predicting POP. Model derivation utilized a registry from Nanjing First Hospital, and external validation was performed using data from patients at the Fourth Affiliated Hospital of Nanjing Medical University. The derivation cohort was divided into the training set and the testing set. The least absolute shrinkage and selection operator (LASSO) and multivariable logistic regression were used for feature screening. We compared the performance of models to select the optimized model and introduced SHapley Additive exPlanations (SHAP) to interpret the model.

Results: The derivation and validation cohorts comprised 498 and 124 patients, with 14.3% and 10.5% POP rates, respectively. Among these models, Categorical boosting (Catboost) demonstrated superior discrimination ability. AUROC was 0.895 (95%CI: 0.841-0.949) and 0.835 (95%CI: 0.740-0.930) on the training and testing sets, respectively. At external validation, the AUROC amounted to 0.894 (95% CI: 0.821-0.966). The SHAP method showed that CRP, the modified five-item frailty index (mFI-5), and ASA body status were among the top three important predicators of POP.

Conclusion: Our model's good early prediction ability, combined with the implementation of a network risk calculator based on the Catboost model, was anticipated to effectively distinguish high-risk POP groups, facilitating timely intervention.

Keywords: Catboost; Orthopedic surgery; Postoperative pneumonia; Prediction model; Risk factor; mFI-5.

Fig. 1

Flow chart of patient enrollment in this study

Fig. 2

Comparison of AUROC and AUPRC curves among LR, RFC, Catboost, XGB, and LGBM in the training and testing sets. (A) AUROC curves of the training set (B) AUROC curves of the testing set (C) AUPRC curves of the training set (D) AUPRC curves of the testing set. AUROC, the area under the receiver operating characteristic; AUPRC, the area under the precision-recall curve; LR, logistic regression; RFC, random forest classifier; Catboost, categorical boosting; XGB, extreme gradient boosting; LGBM, light gradient boosting machine

Fig. 3

Calibration plots for the probability of pneumonia from the five ML models in the training set (A) and the testing set (B). LR, logistic regression; RFC, random forest classifier; Catboost, categorical boosting; XGB, extreme gradient boosting; LGBM, light gradient boosting machine

Fig. 4

The AUROC curves (A), AUPRC curves (B), and calibration plots (C) from the five ML models in the external validation set. LR, logistic regression; RFC, random forest classifier; Catboost, categorical boosting; XGB, extreme gradient boosting; LGBM, light gradient boosting machine

Fig. 5

SHAP summary plot for the seven influential variables in the Catboost model. (A) The average absolute influence of each factor on the model output magnitude was presented in descending order of feature significance; (B) The graph depicted the dot estimate of the Catboost model output, with each dot corresponding to a patient in the dataset. Catboost, categorical boosting; mFI-5, modified five-item frailty index; SpO₂, Peripheral capillary oxygen saturation; ASA, American Society of Anesthesiologists

Fig. 6

The risk web calculator was designed based on the Catboost model. Catboost, categorical boosting