Comprehensive evaluation of composite gene features in cancer outcome prediction

Abstract

Owing to the heterogeneous and continuously evolving nature of cancers, classifiers based on the expression of individual genes usually do not result in robust prediction of cancer outcome. As an alternative, composite gene features that combine functionally related genes have been proposed. It is expected that such features can be more robust and reproducible since they can capture the alterations in relevant biological processes as a whole and may be less sensitive to fluctuations in the expression of individual genes. Various algorithms have been developed for the identification of composite features and inference of composite gene feature activity, which all claim to improve the prediction accuracy. However, because of the limitations of test datasets incorporated by each individual study and inconsistent test procedures, the results of these studies are sometimes conflicting and unproducible. For this reason, it is difficult to have a comprehensive understanding of the prediction performance of composite gene features, particularly across different cancers, cancer subtypes, and cohorts. In this study, we implement various algorithms for the identification of composite gene features and their utilization in cancer outcome prediction, and perform extensive comparison and evaluation using seven microarray datasets covering two cancer types and three different phenotypes. Our results show that, while some algorithms outperform others for certain classification tasks, no single algorithm consistently outperforms other algorithms and individual gene features.

Keywords: cancer; gene expression; outcome prediction; protein interaction networks; systems biology.

Figure 1

Schematic illustration of test process. For each disease and outcome combination, the datasets are matched into pairs. The first dataset in each pair and pathway or PPI data are used for feature identification using various algorithms. The second dataset is used for feature selection, training, and testing using five-fold cross-validation. For this purpose, features extracted from the first dataset are ranked using the training data from the second dataset, based on the P-value of t-test score or other ranking criteria based on discrimination of two phenotype classes. Top 50 features are selected according to these criteria, and SVM and logistic regression classifiers are trained with top K (K = 1, 2,…, 50) features on training data and tested on the testing dataset.

Figure 2

The stability and reproducibility of composite gene features across different datasets. (A) The overlap between the composite gene features identified by each algorithm on two different datasets with the same phenotype. The box plot of Jaccard indices for each algorithm is shown. For each algorithm, feature extraction was performed on five pairs of datasets. Jaccard index was computed for overlap of genes in the top-scoring 50 features for each pair of datasets. (B) The box plot of average t-statistics of top 50 features is shown for each algorithm across seven different datasets. For each dataset, top 50 features are extracted. t-Statistics are calculated with each dataset, and average t-test scores are plotted for these 50 features. (C) The box plot of average t-test statistics of top 50 features for each algorithm on 12 testing datasets. Seven sets of top 50 features from (B) are applied to their paired dataset to compute the average t-statistic on the paired dataset, resulting in 12 data points.

Figure 3

Overall performance of different composite feature identification algorithms. Average of (A) average and (B) maximum AUC values provided by the features identified by each algorithm on 12 test cases. (C) Heat map of relative performance for each test for different algorithms. For each test, relative performance values are calculated as the fraction of average AUC value provided by composite features to the average AUC value provided by individual gene features.

Figure 4

Impact of search criterion on prediction performance. (A) Comparison of mutual information and t-statistic. Genes are ranked based on mutual information computed using GSE2034 dataset and average mutual information, and t-statistics of top 100, 200, …, 1000 genes are plotted. Performance comparison of hybrid algorithms GreedyTtest, LP1-MI, and LP2-MI on test cases (B) GSE2034–GSE7390, (C) GSE17536–GSE17537, and (D) GSE27854–GSE17537.

Figure 5

Performance comparison between aggregate activity and probabilistic inference of feature activity. Average of (A) average and (B) maximum AUC values across 12 test cases for each algorithm is shown for the two different methods used in feature activity inference.

Figure 6

Performance comparison of feature selection algorithms in selecting composite gene features. (A) Average and (B) maximum AUC values of top 50 individual gene features selected with P-value, MRMR, and SVM-RFE for the 12 test cases. (C) Average and (D) maximum AUC values of top 50 GreedyMI features selected with P-value, MRMR, and SVM-RFE for the 12 test cases.

Figure 7

Comparison of forward selection and filter-based feature selection. Performance of (A) the top feature and (B) features selected with forward selection plotted together with average and maximum performance provided by top 50 individual gene features. Performance of (C) the top six features and (D) features selected with forward selection plotted together with average and maximum performance provided by top 50 composite gene features identified by the GreedyMI algorithm.