The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures

Abstract

Biomarker discovery from high-dimensional data is a crucial problem with enormous applications in biology and medicine. It is also extremely challenging from a statistical viewpoint, but surprisingly few studies have investigated the relative strengths and weaknesses of the plethora of existing feature selection methods. In this study we compare 32 feature selection methods on 4 public gene expression datasets for breast cancer prognosis, in terms of predictive performance, stability and functional interpretability of the signatures they produce. We observe that the feature selection method has a significant influence on the accuracy, stability and interpretability of signatures. Surprisingly, complex wrapper and embedded methods generally do not outperform simple univariate feature selection methods, and ensemble feature selection has generally no positive effect. Overall a simple Student's t-test seems to provide the best results.

Figure 1. Area under the ROC curve.

Signature of size in a -fold CV setting and averaged over the four datasets.

Figure 2. Area under the ROC Curve.

NC classifier trained as a function of the size of the signature, for different feature selection methods, in a -fold CV setting averaged over the four datasets.

Figure 3. Area under the ROC Curve.

NC classifier trained as a function of the number of samples in a -fold CV setting. We show here the accuracy for 100-gene signatures as averaged over the datasets. Note that the maximum value of the x axis is constrained by the smallest dataset, namely GSE2990.

Figure 4. Area under the ROC Curve.

NC classifier trained as a function of the number of samples in a -fold CV setting for each of the four datasets. We show here the accuracy for 100-gene signatures.

Figure 5. Stability for a signature of size 100.

Average and standard errors are obtained over the four datasets. a) Soft-perturbation setting. b) Hard-perturbation setting. c) Between-datasets setting.

Figure 6. Evolution of stability of t-test signatures with respect to the size of the training set in the hard-perturbation and the between datasets settings from GSE2034 and GSE4922.

Figure 7. Stability of different methods in the between-dataset setting, as a function of the size of the signature.

Figure 8. GO interpretability for a signature of size 100.

Average number of GO BP terms significantly over-represented.

Figure 9. GO stability for a signature of size 100 in the soft-perturbation setting.

Average and standard errors are obtained over the four datasets. A) Soft-perturbation setting. B) Hard-perturbation setting. C) Between-datasets setting.

Figure 10. Bias in the selection through entropy and Bhattacharyya distance.

Estimated cumulative distribution functions (ECDF) of the first ten genes selected by four methods on GSE1456. They are compared to the ECDF of randomly chosen background genes.

Figure 11. Estimated distribution of the first gene selected by entropy and Bhattacharyya distance.

Figure 12. Accuracy/stability trade-off.

Accuracy versus stability for each method in the between-datasets setting. We show here the average results over the four datasets.