Evaluation of a Machine Learning-Based Prognostic Model for Unrelated Hematopoietic Cell Transplantation Donor Selection

Abstract

The survival of patients undergoing hematopoietic cell transplantation (HCT) from unrelated donors for acute leukemia exhibits considerable variation, even after stringent genetic matching. To improve the donor selection process, we attempted to create an algorithm to quantify the likelihood of survival to 5 years after unrelated donor HCT for acute leukemia, based on the clinical characteristics of the donor selected. All standard clinical variables were included in the model, which also included average leukocyte telomere length of the donor based on its association with recipient survival in severe aplastic anemia, and links to multiple malignancies. We developed a multivariate classifier that assigned a Preferred or NotPreferred label to each prospective donor based on the survival of the recipient. In a previous analysis using a resampling method, recipients with donors labeled Preferred experienced clinically compelling better survival compared with those labeled NotPreferred by the test. However, in a pivotal validation study in an independent cohort of 522 patients, the overall survival of the Preferred and NotPreferred donor groups was not significantly different. Although machine learning approaches have successfully modeled other biological phenomena and have led to accurate predictive models, our attempt to predict HCT outcomes after unrelated donor transplantation was not successful.

Keywords: Acute leukemia; Allogeneic hematopoietic cell transplantation; Donor selection; Leukocyte telomere length; Machine learning.

Figure 1

The process used to define the set of variables and the model used for validation.

Figure 2

A graph of relevant statistics for a large collection of SVM classifiers developed for the HCT donor selection application. Each dot represents a classifier, which labels donors as Preferred (or, equivalently, “POS”, for Positive) and NotPreferred (or, equivalently, “NEG”, for Negative). The x-axis is the proportion of donors labeled Preferred (i.e., “POS”) by the classifier. The y-axis is the survival benefit (difference in survival at 5 years) conferred by the donors, compared with survival of recipients who received HCT from NotPreferred donors. A clinically attractive classifier, selected for the validation, is labeled by red arrow. It is defined as the classifier which maximizes clinical benefit while labeling at least 10% of donors as Preferred.

Figure 3

Survival of recipients of donors labeled Preferred and NotPreferred. The graph was produced using ten-fold cross- validation. HR = 0.43 (95% CI, 0.28 to 0.67), log-rank P < 0.001.

Figure 4

Survival of recipients of donors labeled Poor and NotPoor by the less stringent model, in ten-fold cross-validation. HR = 0.75 (95% CI, 0.61 to 0.91), log-rank P = 0.003.

Figure 5

Validation KM graph for the primary classification model. HR = 1.12 (95% CI, 0.72 to 1.72), log-rank P = 0.62.

Figure 6

Exploratory model validation results at five years. HR = 1.18 (95% CI, 0.94 to 1.48), log-rank P = 0.148.

Figure 7A and 7B

Primary classification model validation results for AML and ALL patients, respectively. AML HR = 2.01 (95% CI, 1.22 to 3.3), log-rank P = 0.005. ALL HR = 0.42 (95% CI, 0.17 to 1.02), log-rank P = 0.049.

Figure 8A and 8B

Primary classification model training (cross-validation) results for AML and ALL patients, respectively. AML HR = 0.56 (95% CI, 0.28 to 1.09), log-rank P = 0.083. ALL HR = 0.37 (95% CI, 0.21 to 0.66), log-rank P < 0.001.