Ranking metrics in gene set enrichment analysis: do they matter?

Abstract

Background: There exist many methods for describing the complex relation between changes of gene expression in molecular pathways or gene ontologies under different experimental conditions. Among them, Gene Set Enrichment Analysis seems to be one of the most commonly used (over 10,000 citations). An important parameter, which could affect the final result, is the choice of a metric for the ranking of genes. Applying a default ranking metric may lead to poor results.

Methods and results: In this work 28 benchmark data sets were used to evaluate the sensitivity and false positive rate of gene set analysis for 16 different ranking metrics including new proposals. Furthermore, the robustness of the chosen methods to sample size was tested. Using k-means clustering algorithm a group of four metrics with the highest performance in terms of overall sensitivity, overall false positive rate and computational load was established i.e. absolute value of Moderated Welch Test statistic, Minimum Significant Difference, absolute value of Signal-To-Noise ratio and Baumgartner-Weiss-Schindler test statistic. In case of false positive rate estimation, all selected ranking metrics were robust with respect to sample size. In case of sensitivity, the absolute value of Moderated Welch Test statistic and absolute value of Signal-To-Noise ratio gave stable results, while Baumgartner-Weiss-Schindler and Minimum Significant Difference showed better results for larger sample size. Finally, the Gene Set Enrichment Analysis method with all tested ranking metrics was parallelised and implemented in MATLAB, and is available at https://github.com/ZAEDPolSl/MrGSEA .

Conclusions: Choosing a ranking metric in Gene Set Enrichment Analysis has critical impact on results of pathway enrichment analysis. The absolute value of Moderated Welch Test has the best overall sensitivity and Minimum Significant Difference has the best overall specificity of gene set analysis. When the number of non-normally distributed genes is high, using Baumgartner-Weiss-Schindler test statistic gives better outcomes. Also, it finds more enriched pathways than other tested metrics, which may induce new biological discoveries.

Keywords: Functional genomics; GSEA; Pathway analysis; Ranking metrics.

Fig. 1

Boxplots of surrogate sensitivity and FPR of gene set analysis. Panel a represents the distribution of target pathways enrichment p-value to each metric presented in logarithmic scale - the lower the better; Panel b represents the results of FPR estimation, where the red line represents the expected outcome - the closer to 5% the better

Fig. 2

Results of k-means cluster analysis based on three performance criteria. Results highlighted with green colour show good performance, red colour represents poor performance and yellow colour represents medium performance

Fig. 3

Results of k-means cluster analysis based on two performance criteria. The best results have those metrics, where FPR estimation is closest to 0, and sensitivity estimation (1-${\widehat{\pi }}_0$) is closest to 1

Fig. 4

Robustness of ranking metrics to sample size. Panel a represents surrogate sensitivity assessment of four best metrics for different sample size. Panel b represents FPR estimates under tested sample size

Fig. 5

Results of detecting significant gene sets across various thresholds. Panel a represents percentage of significantly enriched pathways. Solid lines represent average value across analysed data sets whereas dashed lines represent its confidence intervals. Panel b represents percentage of significantly enriched pathways in experiment design dedicated to FPR evaluation. Red dashed line represents the expected outcome