External validation of the Bayesian Estimated Tools for Survival (BETS) models in patients with surgically treated skeletal metastases

Abstract

Background: We recently developed two Bayesian networks, referred to as the Bayesian-Estimated Tools for Survival (BETS) models, capable of estimating the likelihood of survival at 3 and 12 months following surgery for patients with operable skeletal metastases (BETS-3 and BETS-12, respectively). In this study, we attempted to externally validate the BETS-3 and BETS-12 models using an independent, international dataset.

Methods: Data were collected from the Scandinavian Skeletal Metastasis Registry for patients with extremity skeletal metastases surgically treated at eight major Scandinavian referral centers between 1999 and 2009. These data were applied to the BETS-3 and BETS-12 models, which generated a probability of survival at 3 and 12 months for each patient. Model robustness was assessed using the area under the receiver-operating characteristic curve (AUC). An analysis of incorrect estimations was also performed.

Results: Our dataset contained 815 records with adequate follow-up information to establish survival at 12 months. All records were missing data including the surgeon's estimate of survival, which was previously shown to be a first-degree associate of survival in both models. The AUCs for the BETS-3 and BETS-12 models were 0.79 and 0.76, respectively. Incorrect estimations by both models were more commonly optimistic than pessimistic.

Conclusions: The BETS-3 and BETS-12 models were successfully validated using an independent dataset containing missing data. These models are the first validated tools for accurately estimating postoperative survival in patients with operable skeletal metastases of the extremities and can provide the surgeon with valuable information to support clinical decisions in this patient population.

Figure 1

BETS-3 model structure. As shown, there are 5 first-degree associates of 3-month survival: surgeon’s estimate of survival, preoperative hemoglobin concentration, preoperative absolute lymphocyte count, ECOG performance status, and presence of a complete pathologic fracture.

Figure 2

BETS-12 model structure. As shown, there are four first-degree associates of 12-month survival: surgeon’s estimate of survival, preoperative hemoglobin concentration, number of bone metastases, and primary oncologic diagnosis.

Figure 3

Overlay plot of classifications made by the BETS-3 model. This plot illustrates which records were correctly (green dot) and incorrectly (red ×) classified as a function of the model’s predicted (estimated) probability of survival greater than 3 months. Most misclassifications were optimistic, with a median estimated probability of 0.64 (total range 0.06-1.00; interquartile range 0.44, 0.83). Three-month survival was overestimated in 142 records (incorrectly classified records from probability 0.5 to 1). In these cases, patients did not live as long as the estimated 3 months and surgery, performed at the end of life, may have been unnecessary.

Figure 4

Overlay plot of classifications made by the BETS-12 model. This plot illustrates which records were correctly (green dot) and incorrectly (red ×) classified as a function of the model’s predicted (estimated) probability of survival greater than 12 months. Most misclassifications were optimistic, with a median estimated probability of 0.60 (total range 0.26-0.96; interquartile range 0.51, 0.76). Twelve-month survival was underestimated in 62 records (incorrectly classified records from probability 0 to 0.5) in which patients lived longer than the estimated 12 months. They represent cases at risk for implant failure if less-invasive or less durable constructs are used.