New developments for the Quest for Orthologs benchmark service

Abstract

The Quest for Orthologs (QfO) orthology benchmark service (https://orthology.benchmarkservice.org) hosts a wide range of standardized benchmarks for orthology inference evaluation. It is supported and maintained by the QfO consortium, and is used to gather ortholog predictions and to examine strengths and weaknesses of newly developed and existing orthology inference methods. The web server allows different inference methods to be compared in a standardized way using the same proteome data. The benchmark results are useful for developing new methods and can help researchers to guide their choice of orthology method for applications in comparative genomics and phylogenetic analysis. We here present a new release of the Orthology Benchmark Service with a new benchmark based on feature architecture similarity as well as updated reference proteomes. We further provide a meta-analysis of the public predictions from 18 different orthology assignment methods to reveal how they relate in terms of ortholog predictions and benchmark performance. These results can guide users of orthologs to the best suited method for their purpose.

Graphical Abstract

Figure 1.

Feature architecture comparison as a novel benchmark in the QfO benchmark service. (A) Feature architecture comparison of an ortholog pair that was consistently found by all methods. The average bi-directional FAS score is 0.85 due to the Leucine-rich region that is present only in the human protein. (B) Feature architecture comparison of an ortholog pair that was assigned only by SonicParanoid. The two proteins differ substantially in both their length and in their feature architecture with the sole feature being shared is the C-terminal UDPGT Pfam domain. The average bi-directional FAS score is 0.43. (C) The correlation between the mean FAS score of protein pairs and the number of ortholog predictors supporting the orthology relationship (Pearson’s correlation coefficient: 0.98, P = 6e-12). (D) FAS benchmark performance versus the number of inferred orthologs for orthology assignment tools submitted to the latest QfO orthology benchmark service.

Figure 2.

Orthologous pairs inferred by the 18 public methods in the benchmarking service. Subsections of the bars represent the number of methods that share the same pairs, including the method in question. Green parts of the bars are unique to the method. Methods are ranked by the number of pairs they share with at least one other method (non-green part of the stacked bars).

Figure 3.

Pairwise representation of overlap between orthology inference methods included in the QfO Benchmarking service. The heatmap shows the proportion of the pairs inferred by methods on the right side that are found by methods on the bottom. The heatmap is hierarchically clustered on rows and columns by similarity with the corresponding trees shown.

Figure 4.

Relatedness of orthologous pair predictions between the orthology inference methods included in the QfO Benchmarking service by PCA. The inset shows an enlarged picture of the cluster that contains all methods except Ensembl Compara and OMA HOGs.

Figure 5.

Relatedness of benchmark performance between the orthology inference methods included in the QfO Benchmarking service by PCA.

Figure 6.

Paralogs of human protein ABCA1, as shown on the Alliance of Genome Resources website. Paralogs are ordered by protein sequence similarity (including pairwise alignment length and % amino acid similarity) as well as agreement across different QfO member resources. The URL for this specific table is https://www.alliancegenome.org/gene/HGNC:29#paralogy.