Development of Machine Learning Algorithms Incorporating Electronic Health Record Data, Patient-Reported Outcomes, or Both to Predict Mortality for Outpatients With Cancer

Abstract

Purpose: Machine learning (ML) algorithms that incorporate routinely collected patient-reported outcomes (PROs) alongside electronic health record (EHR) variables may improve prediction of short-term mortality and facilitate earlier supportive and palliative care for patients with cancer.

Methods: We trained and validated two-phase ML algorithms that incorporated standard PRO assessments alongside approximately 200 routinely collected EHR variables, among patients with medical oncology encounters at a tertiary academic oncology and a community oncology practice.

Results: Among 12,350 patients, 5,870 (47.5%) completed PRO assessments. Compared with EHR- and PRO-only algorithms, the EHR + PRO model improved predictive performance in both tertiary oncology (EHR + PRO v EHR v PRO: area under the curve [AUC] 0.86 [0.85-0.87] v 0.82 [0.81-0.83] v 0.74 [0.74-0.74]) and community oncology (area under the curve 0.89 [0.88-0.90] v 0.86 [0.85-0.88] v 0.77 [0.76-0.79]) practices.

Conclusion: Routinely collected PROs contain added prognostic information not captured by an EHR-based ML mortality risk algorithm. Augmenting an EHR-based algorithm with PROs resulted in a more accurate and clinically relevant model, which can facilitate earlier and targeted supportive care for patients with cancer.

FIG 1.

The correlations between the PROs in the data set. Darker and larger dots indicate stronger correlations. For example, the correlation between decreased performance status and fatigue was 0.69, while the correlation between anxiety and sadness was 0.72. PRO, patient-reported outcome.

FIG 2.

Univariable associations between PROs and 180-day mortality. 180-day mortality was defined as a binary indicator variable. PROs were coded on a 1-5 Likert scale, with greater values indicating more severe symptoms, with the exception of rash, which was coded on a 0-1 scale (present/absent). PAH, Pennsylvania Hospital; PCAM, Perelman School of Advanced Medicine; PRO, patient-reported outcome.

FIG 3.

Comparison of model performance metrics between the EHR + PRO, EHR, and PRO algorithms at tertiary oncology (PCAM) and community oncology (PAH) practices. Model performance metrics include (A) AUC, (B) AUPRC, (C) TPR, and (D) FPR. TPR and FPR were calculated using a 10% mortality risk threshold, which corresponds to the risk threshold currently used in clinical practice. AUC, area under the curve; AUPRC, area under the precision-recall curve; EHR, electronic health record; FPR, false-positive rate; PAH, Pennsylvania Hospital; PCAM, Perelman School of Advanced Medicine; PRO, patient-reported outcome; TPR, true-positive rate.

FIG 4.

Decision curve analysis showing standardized net benefit of EHR + PRO, EHR, and PRO algorithms across several model risk thresholds at the tertiary oncology practice (PCAM). A standardized decision curve plots the net benefit to the population of using a risk model against a range of risk threshold values for identifying high-risk patients. A risk threshold chosen to assess the net benefit reflects the user's perspective on the relative cost of a false-positive and false-negative prediction of patients' high-risk status. The standardized net benefit is defined as the difference between the true-positive and the weighted false-positive rates, where the weight is calculated as the odds of the risk threshold multiplied by the inverse odds of the outcome prevalence. It has a maximum of one and can be interpreted as the fraction of maximum utility achieved by the model at the given risk threshold where the maximum utility is achieved when TPR = 1 and FPR = 0. EHR, electronic health record; FPR, false-positive rate; PCAM, Perelman School of Advanced Medicine; PRO, patient-reported outcome; TPR, true-positive rate.

FIG A1.

Cohort selection criteria. PAH, Pennsylvania Hospital; PCAM, Perelman School of Advanced Medicine.

FIG A2.

Association between PRO trends and mortality. PRO, patient-reported outcome.

FIG A3.

Associations between PROs at tertiary oncology practice. PRO, patient-reported outcome.

FIG A4.

Associations between PROs at community oncology practice. PRO, patient-reported outcome.

FIG A5.

PROs included in final model and associations with mortality. EHR, electronic health record; PAH, Pennsylvania Hospital; PCAM, Perelman School of Advanced Medicine; PRO, patient-reported outcome.