External Validation of PATHFx Version 3.0 in Patients Treated Surgically and Nonsurgically for Symptomatic Skeletal Metastases

Abstract

Background: PATHFx is a clinical decision-support tool based on machine learning capable of estimating the likelihood of survival after surgery for patients with skeletal metastases. The applicability of any machine-learning tool depends not only on successful external validation in unique patient populations but also on remaining relevant as more effective systemic treatments are introduced. With advancements in the treatment of metastatic disease, it is our responsibility to patients to ensure clinical support tools remain contemporary and accurate.

Question/purposes: Therefore, we sought to (1) generate updated PATHFx models using recent data from patients treated at one large, urban tertiary referral center and (2) externally validate the models using two contemporary patient populations treated either surgically or nonsurgically with external-beam radiotherapy alone for symptomatic skeletal metastases for symptomatic lesions.

Methods: After obtaining institutional review board approval, we collected data on 208 patients undergoing surgical treatment for pathologic fractures at Memorial Sloan Kettering Cancer Center between 2015 and 2018. These data were combined with the original PATHFx training set (n = 189) to create the final training set (n = 397). We then created six Bayesian belief networks designed to estimate the likelihood of 1-month, 3-month, 6-month, 12-month, 18-month, and 24-month survival after treatment. Bayesian belief analysis is a statistical method that allows data-driven learning to arise from conditional probabilities by exploring relationships between variables to estimate the likelihood of an outcome using observed data. For external validation, we extracted the records of patients treated between 2016 and 2018 from the International Bone Metastasis Registry and records of patients treated nonoperatively with external-beam radiation therapy for symptomatic skeletal metastases from 2012 to 2016 using the Military Health System Data Repository (radiotherapy-only group). From each record, we collected the date of treatment, laboratory values at the time of treatment initiation, demographic data, details of diagnosis, and the date of death. All records reported sufficient follow-up to establish survival (yes/no) at 24-months after treatment. For external validation, we applied the data from each record to the new PATHFx models. We assessed calibration (calibration plots), accuracy (Brier score), discriminatory ability (area under the receiver operating characteristic curve [AUC]).

Results: The updated PATHFx version 3.0 models successfully classified survival at each time interval in both external validation sets and demonstrated appropriate discriminatory ability and model calibration. The Bayesian models were reasonably calibrated to the Memorial Sloan Kettering Cancer Center training set. External validation with 197 records from the International Bone Metastasis Registry and 192 records from the Military Health System Data Repository for analysis found Brier scores that were all less than 0.20, with upper bounds of the 95% confidence intervals all less than 0.25, both for the radiotherapy-only and International Bone Metastasis Registry groups. Additionally, AUC estimates were all greater than 0.70, with lower bounds of the 95% CI all greater than 0.68, except for the 1-month radiotherapy-only group. To complete external validation, decision curve analysis demonstrated clinical utility. This means it was better to use the PATHFx models when compared to the default assumption that all or no patients would survive at all time periods except for the 1-month models. We believe the favorable Brier scores (< 0.20) as well as DCA indicate these models are suitable for clinical use.

Conclusions: We successfully updated PATHFx using contemporary data from patients undergoing either surgical or nonsurgical treatment for symptomatic skeletal metastases. These models have been incorporated for clinical use on PATHFx version 3.0 (https://www.pathfx.org). Clinically, external validation suggests it is better to use PATHFx version 3.0 for all time periods except when deciding whether to give radiotherapy to patients with the life expectancy of less than 1 month. This is partly because most patients survived 1-month after treatment. With the advancement of medical technology in treatment and diagnosis for patients with metastatic bone disease, part of our fiduciary responsibility is to the main current clinical support tools.

Level of evidence: Level III, therapeutic study.

Fig. 1

A-F The calibration curves of validation set data show the agreement between observed outcomes and those predicted by the (A) 1-month, (B) 3-month, (C) 6-month, (D) 12-month, (E) 18-month, and (F) 24-month PATHFx models. The shaded region depicts the 95% CI of the predictions. Perfect calibration to the training data overlies the 45° dotted line. Each model is reasonably well calibrated to the Memorial Sloan Kettering Cancer Center training data.

Fig. 2

A-F The decision curve analysis (dashed line) of the predictive models based on International Bone Metastasis Registry data at (A) 1-month, (B) 3-months, (C) 6-months, (D) 12-months, (E) 18 months, and (F) 24 months indicates that all the models should be used rather than assuming all patients (continuous line) or none of the patients (thick continuous line) will survive longer than the period of each predictive model.

Fig. 3

A-F The decision curve analysis (dashed line) of the predictive models based on RT data at (A) 1 month, (B) 3 months, (C) 6 months, (D) 12 months, (E) 18 months, and (F) 24 months indicates that all the models should be used rather than assuming all patients (continuous line) or none of the patients (thick continuous line) will survive longer than the period of each predictive model, except for the 1-month model where it is better to assume all survive than use the model; RT = radiotherapy-only group.