Systematic comparison of ranking aggregation methods for gene lists in experimental results

Abstract

Motivation: A common experimental output in biomedical science is a list of genes implicated in a given biological process or disease. The gene lists resulting from a group of studies answering the same, or similar, questions can be combined by ranking aggregation methods to find a consensus or a more reliable answer. Evaluating a ranking aggregation method on a specific type of data before using it is required to support the reliability since the property of a dataset can influence the performance of an algorithm. Such evaluation on gene lists is usually based on a simulated database because of the lack of a known truth for real data. However, simulated datasets tend to be too small compared to experimental data and neglect key features, including heterogeneity of quality, relevance and the inclusion of unranked lists.

Results: In this study, a group of existing methods and their variations that are suitable for meta-analysis of gene lists are compared using simulated and real data. Simulated data were used to explore the performance of the aggregation methods as a function of emulating the common scenarios of real genomic data, with various heterogeneity of quality, noise level and a mix of unranked and ranked data using 20 000 possible entities. In addition to the evaluation with simulated data, a comparison using real genomic data on the SARS-CoV-2 virus, cancer (non-small cell lung cancer) and bacteria (macrophage apoptosis) was performed. We summarize the results of our evaluation in a simple flowchart to select a ranking aggregation method, and in an automated implementation using the meta-analysis by information content algorithm to infer heterogeneity of data quality across input datasets.

Availability and implementation: The code for simulated data generation and running edited version of algorithms: https://github.com/baillielab/comparison_of_RA_methods. Code to perform an optimal selection of methods based on the results of this review, using the MAIC algorithm to infer the characteristics of an input dataset, can be downloaded here: https://github.com/baillielab/maic. An online service for running MAIC: https://baillielab.net/maic.

Supplementary information: Supplementary data are available at Bioinformatics online.

Fig. 1.

(a) An overview of the methods for this study. (b) Investigated ranking aggregation methods in this study and some key features of them. Distribution based: the method is based on the distribution of latent model or calculated statistics (Li et al., 2019). Stochastic: the method includes a stochastic process like random sampling. Weighting lists: the method assigns weightings to lists explicitly to show their difference, like quantifying the quality. Simple statistics: use simple statistics, like frequency or average ranking. Bayesian methods are labeled using underline. Whether the methods are designed to only take ranked lists as the input or be able to accept unranked lists are also marked

Fig. 2.

Results for simulated datasets and real datasets. All subplots use the same color and line styles to show investigated methods. Detailed results can be seen in Supplementary File S1 and Supporting data. (a–d) Results for simulated data with various mean noise levels and quality heterogeneity. The mean of accuracy using top-1000 cutoff (except for the right figure of d which uses top-100 cutoff) and 95% confidence interval are plotted for 100 repeated experiments using lines and shading separately. The default setting of absent gene rate γ = 0 is used. The simulated dataset type shows properties for datasets. The first part shows whether a dataset includes a mix of ranked and unranked sources (Mix) or only includes ranked sources (Rank). The second part shows the number of included sources (large or small). (e) Real data: results for the collected SARS-CoV-2, macrophages apoptosis and NSCLC datasets. The recall with cutoffs from top-1 to top-1000 is shown

Fig. 3.

A flowchart for selecting methods depending on the ranking information, the number of sources included and the heterogeneity of quality for the investigated sources, generated following the evaluation result of this study. Multiple methods within the same block means they perform similarly with the best performance under the corresponding scenario