Computerized Mortality Prediction for Community-acquired Pneumonia at 117 Veterans Affairs Medical Centers

Abstract

Rationale: Computerized severity assessment for community-acquired pneumonia could improve consistency and reduce clinician burden. Objectives: To develop and compare 30-day mortality-prediction models using electronic health record data, including a computerized score with all variables from the original Pneumonia Severity Index (PSI) except confusion and pleural effusion ("ePSI score") versus models with additional variables. Methods: Among adults with community-acquired pneumonia presenting to emergency departments at 117 Veterans Affairs Medical Centers between January 1, 2006, and December 31, 2016, we compared an ePSI score with 10 novel models employing logistic regression, spline, and machine learning methods using PSI variables, age, sex and 26 physiologic variables as well as all 69 PSI variables. Models were trained using encounters before January 1, 2015; tested on encounters during and after January 1, 2015; and compared using the areas under the receiver operating characteristic curve, confidence intervals, and patient event rates at a threshold PSI score of 970. Results: Among 297,498 encounters, 7% resulted in death within 30 days. When compared using the ePSI score (confidence interval [CI] for the area under the receiver operating characteristic curve, 0.77-0.78), performance increased with model complexity (CI for the logistic regression PSI model, 0.79-0.80; CI for the boosted decision-tree algorithm machine learning PSI model using the Extreme Gradient Boosting algorithm [mlPSI] with the 19 original PSI factors, 0.83-0.85) and the number of variables (CI for the logistic regression PSI model using all 69 variables, 0.84-085; CI for the mlPSI with all 69 variables, 0.86-0.87). Models limited to age, sex, and physiologic variables also demonstrated high performance (CI for the mlPSI with age, sex, and 26 physiologic factors, 0.84-0.85). At an ePSI score of 970 and a mortality-risk cutoff of <2.7%, the ePSI score identified 31% of all patients as being at "low risk"; the mlPSI with age, sex, and 26 physiologic factors identified 53% of all patients as being at low risk; and the mlPSI with all 69 variables identified 56% of all patients as being at low risk, with similar rates of mortality, hospitalization, and 7-day secondary hospitalization being determined. Conclusions: Computerized versions of the PSI accurately identified patients with pneumonia who were at low risk of death. More complex models classified more patients as being at low risk of death and as having similar adverse outcomes.

Keywords: clinical prediction models; decision support; machine learning; pneumonia.

Figure 1.

Study population. VA = Veterans Affairs.

Figure 2.

Performance of models in terms of 30-day mortality prediction: comparison of all models in the validation cohort (67,028 encounters). Performance, measured by using the AUROC, is shown for the ePSI and 10 novel models and is ranked by the complexity of the model (logistic, spline, and machine learning). AUROC = area under the receiver operating characteristic curve; ePSI score = computerized score with all variables from the original PSI except confusion and pleural effusion; Logistic-PSI = logistic regression PSI model; Logistic-PSI-28 = Logistic-PSI with age, sex, and 26 physiologic factors; Logistic-PSI-69 = Logistic-PSI with all 69 variables; Logistic-PSI-continuous = Logistic-PSI including the PSI factors with continuous variables and assuming a linear relationship; Logisitic-PSI-re-weighted = Logistic-PSI refit with PSI factors and the original binary cutoffs; ml-PSI = boosted decision-tree algorithm machine learning PSI model using the Extreme Gradient Boosting algorithm; ml-PSI-28 = ml-PSI with age, sex, and 26 physiologic factors; ml-PSI-69 = ml-PSI with all 69 variables; PSI = Pneumonia Severity Index; Spline-PSI = spline Logistic-PSI; Spline-PSI-28 = Spline-PSI with age, sex, and 26 physiologic factors; Spline-PSI-69 = Spline-PSI with 69 total patient factors.

Figure 3.

Performance of models in terms of 30-day mortality prediction: comparison of the ePSI score with machine learning models in the validation cohort (N = 67,028). (A) Overall predictive performance was examined by plotting receiver operating characteristic curves. (B) Calibration was evaluated by plotting the observed versus the expected 30-day mortality rate among patients at each vigintile of risk predicted by each model. (C) Discrimination was examined by plotting the distribution/density of events against the predicted risk, stratified by outcome (dashed line represents perfect calibration). (D) Generalizability was examined by calculating the AUROC for each subgroup and year. AUROC = area under the receiver operating characteristic curve; ePSI score = computerized score with all variables from the original PSI except confusion and pleural effusion; ml-PSI = boosted decision-tree algorithm machine learning PSI model using the Extreme Gradient Boosting algorithm; ml-PSI-28 = ml-PSI with age, sex, and 26 physiologic factors; ml-PSI-69 = ml-PSI with all 69 variables; PSI = Pneumonia Severity Index.

Figure 4.

Facility distribution of the area under the receiver operating characteristic curve (AUC) for the boosted decision-tree algorithm machine learning Pneumonia Severity Index (PSI) model using the Extreme Gradient Boosting algorithm (mlPSI) to predict 30-day mortality across 117 Veterans Affairs facilities. The AUC is plotted against the number of outcome events (30-d death) by the facility for ePSI score, mlPSI, mlPSI-28, and mlPSI-69. The solid line in each plot represents the overall AUC for each model. ePSI score = computerized score with all variables from the original PSI except confusion and pleural effusion; mlPSI-28 = ml-PSI with age, sex, and 26 physiologic factors; mlPSI-69 = mlPSI with all 69 variables.