Independent validation of a model using cell line chemosensitivity to predict response to therapy

Abstract

Background: Methods using cell line microarray and drug sensitivity data to predict patients' chemotherapy response are appealing, but groups may be reluctant to release details to preserve intellectual property. Here we describe a case study to validate predictions while treating the methods as a "black box."

Methods: Medical Prognosis Institute (MPI) constructed cell-line-derived sensitivity scores (SSs) and combined scores (CSs) that incorporate clinical variables. MD Anderson researchers evaluated their predictions. We searched the Gene Expression Omnibus (GEO) to identify validation datasets, and we performed statistical evaluation of the agreement between prediction and clinical observation.

Results: We identified 3 suitable datasets: GSE16446 (n = 120; binary outcome), GSE17920 (n = 130; binary outcome), and GSE10255 (n = 161; continuous and time-to-event outcomes). The SS was statistically significantly associated with primary treatment responses for all studies (GSE16446: P = .02; GSE17920: P = .02; GSE10255: P = .02). Dichotomized SSs performed no better than chance for GSE16446 and GSE17920, and categorized SSs did not predict disease-free survival (GSE10255). SSs sometimes improved on predictions using clinical variables (GSE16446: P = .05; GSE17920: P = .31; GSE10255: P = .045), but gains were limited (95% confidence intervals for GSE16446 and GSE17920 include 0). The CS did not predict treatment response for GSE16446 (P = .55), but it did for GSE17920 (P < .001). Coefficients of clinical variables provided by MPI for CSs agree with estimates for GSE17920 better than estimates for GSE16446.

Conclusions: Model predictions were better than chance in all three datasets. However, these scores added little to existing clinical predictors; statistically significant contributions were likely to be too small to change clinical practice. These findings suggest that discovering better predictors will require both cell line data and a clinical training dataset of patient samples.

Figure 1.

Receiver operating characteristic (ROC) curves of combined score and sensitivity score to predict pathological complete response and 95% confidence intervals of paired (false-positive rate [FPR], true-positive rate [TPR]) and (positive predictive value [PPV], negative predictive value [NPV]) for sensitivity score with two different cutoff points for study GSE16446 (breast cancer). A) Ninety-five percent confidence intervals for (FPR, TPR) sensitivity score. The solid black line and orange line are the ROC curves for sensitivity score and combined score, respectively. B) Ninety-five percent confidence intervals for (PPV, NPV) sensitivity score. Cutoff points were 1) the 86th percentile (1 − the pathological complete response rate of GSE16446 study; red ellipses) and 2) the mean (the standard cutoff point used by the Medical Prognosis Institute; blue ellipses). Gray area indicates the region for the prediction made by chance. AUC = area under the curve; CI = confidence interval.

Figure 2.

Association between the Medical Prognostic Institute (MPI) sensitifit score and outcome in study GSE10255 (acute lymphoblastic leukemia). A) Scatter plot of sensitivity score versus change in the levels of circulating leukemia cells at day 3 after the start of treatment () and the fitted line by linear regression. B) Kaplan-Meier plots of disease-free survival (DFS) categorized by MPI sensitivity score. A good responder was a patient with top 25% sensitivity scores (n = 20). An intermediate responder was a atient with middle 50% sensitivity scores (n = 53). A poor responder was a patient with bottom 25% sensitivity scores (n = 19).