The CAFA challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens

Abstract

Background: The Critical Assessment of Functional Annotation (CAFA) is an ongoing, global, community-driven effort to evaluate and improve the computational annotation of protein function.

Results: Here, we report on the results of the third CAFA challenge, CAFA3, that featured an expanded analysis over the previous CAFA rounds, both in terms of volume of data analyzed and the types of analysis performed. In a novel and major new development, computational predictions and assessment goals drove some of the experimental assays, resulting in new functional annotations for more than 1000 genes. Specifically, we performed experimental whole-genome mutation screening in Candida albicans and Pseudomonas aureginosa genomes, which provided us with genome-wide experimental data for genes associated with biofilm formation and motility. We further performed targeted assays on selected genes in Drosophila melanogaster, which we suspected of being involved in long-term memory.

Conclusion: We conclude that while predictions of the molecular function and biological process annotations have slightly improved over time, those of the cellular component have not. Term-centric prediction of experimental annotations remains equally challenging; although the performance of the top methods is significantly better than the expectations set by baseline methods in C. albicans and D. melanogaster, it leaves considerable room and need for improvement. Finally, we report that the CAFA community now involves a broad range of participants with expertise in bioinformatics, biological experimentation, biocuration, and bio-ontologies, working together to improve functional annotation, computational function prediction, and our ability to manage big data in the era of large experimental screens.

Keywords: Biofilm; Community challenge; Critical assessment; Long-term memory; Protein function prediction.

Fig. 1

A comparison in F_max between the top 5 CAFA2 models against the top 5 CAFA3 models. Colored boxes encode the results such that (1) the colors indicate margins of a CAFA3 method over a CAFA2 method in F_max and (2) the numbers in the box indicate the percentage of wins. a CAFA2 top 5 models (rows, from top to bottom) against CAFA3 top 5 models (columns, from left to right). b Comparison of the performance (F_max) of Naïve baselines trained respectively on SwissProt2014 and SwissProt2017. Colored box between the two bars shows the percentage of wins and margin of wins as in a. c Comparison of the performance (F_max) of BLAST baselines trained on SwissProt2014 and SwissProt2017. Colored box between the two bars shows the percentage of wins and margin of wins as in a. Statistical significance was assessed using 10,000 bootstrap samples of benchmark proteins

Fig. 2

Performance evaluation based on the F_max for the top CAFA1, CAFA2, and CAFA3 methods. The top 12 methods are shown in this barplot ranked in descending order from left to right. The baseline methods are appended to the right; they were trained on training data from 2017, 2014, and 2011, respectively. Coverage of the methods were shown as text inside the bars. Coverage is defined as the percentage of proteins in the benchmark that are predicted by the methods. Color scheme: CAFA2, ivory; CAFA3, green; Naïve, red; BLAST, blue. Note that in MFO and BPO, CAFA1 methods were ranked, but since none made to the top 12 of all 3 CAFA challenges, they were not displayed. The CAFA1 challenge did not collect predictions for CCO. a: molecular function; b: Biological process; c: Cellular Component

Fig. 3

Performance evaluation based on the F_max for the top-performing methods in 3 ontologies. Evaluation was carried out on No knowledge benchmarks in the full mode. a–c: bar plots showing the F_max of the top 10 methods. The 95% confidence interval was estimated using 10,000 bootstrap iterations on the benchmark set. Coverage of the methods was shown as text inside the bars. Coverage is defined as the percentage of proteins in the benchmark which are predicted by the methods. d–f: precision-recall curves for the top 10 methods. The perfect prediction should have F_max=1, at the top right corner of the plot. The dot on the curve indicates where the maximum F score is achieved

Fig. 4

Performance evaluation based on S_min for the top-performing methods in 3 ontologies. Evaluation was carried out on No knowledge benchmarks in the full mode. a–c: bar plots showing S_min of the top 10 methods. The 95% confidence interval was estimated using 10,000 bootstrap iterations on the benchmark set. Coverage of the methods was shown as text inside the bars. Coverage is defined as the percentage of proteins in the benchmark which are predicted by the methods. d–f: remaining uncertainty-missing information (RU-MI) curves for the top 10 methods. The perfect prediction should have S_min=0, at the bottom left corner of the plot. The dot on the curve indicates where the minimum semantic distance is achieved

Fig. 5

Evaluation based on the F_max for the top-performing methods in eukaryotic and bacterial species

Fig. 6

Number of proteins in each benchmark species and ontology

Fig. 7

Heatmap of similarity for the top 10 methods in CAFA1, CAFA2, and CAFA3. Similarity is represented by Euclidean distance of the prediction scores from each pair of methods, using the intersection set of benchmarks in the “Top methods have improved from CAFA2 to CAFA3, but improvement was less dramatic than from CAFA1 to CAFA2” section. The higher (darker red color) the euclidean distance, the less similar the methods are. Top 10 methods from each of the CAFA challenges are displayed and ranked by their performance in F_max. Cells highlighted by black borders are between a pair of methods that come from the same PI. a: Molecular Function; b: Biological Process; c: Cellular Component

Fig. 8

Keyword analysis of all CAFA3 participating methods. a–c: both relative frequency of the keywords and weighted frequencies are provided for three respective GO ontologies. The weighted frequencies accounts for the performance of the the particular model using the given keyword. If that model performs well (with high F_max), then it gives more weight to the calculation of the total weighted average of that keyword. d shows the ratio of relative frequency between the F_max-weighted and equal-weighted. Red indicates the ratio is greater than one while blue indicates the ratio is less than one. Only the top five keywords ranked by ratio are shown. The larger the ratio, the more difference there is between the F_max-weighted and the equal-weighted

Fig. 9

AUROC of the top five teams in CAFA- π. The best-performing model from each team is picked for the top five teams, regardless of whether that model is submitted as model 1. Four baseline models all based on BLAST were computed for Candida, while six baseline models were computed for Pseudomonas, including two based on expression profiles. All team methods are in gray while BLAST methods are in red, BLAST computational methods are in blue, and expression are in yellow, see Table 3 for the description of the baselines

Fig. 10

AUROC of top five teams in CAFA3. The best-performing model from each team is picked for the top five teams, regardless of whether that model is submitted as model 1. All team methods are in gray while BLAST methods are in red and BLAST computational methods are in blue, see Table 3 for the description of the baselines

Fig. 11

CAFA participation has been growing. Each principal investigator is allowed to head multiple teams, but each member can only belong to one team. Each team can submit up to three models

Fig. 12

CAFA3 timeline

Fig. 13

Experimental procedure of determining genes associated with the functions biofilm formation (a) and motility (b) in P. aeruginosa

Fig. 14

a: different phenotypes in response to doxycycline treatment: low growth, smooth, no growth and intermediate. b: adherence phenotypes. See text for details

Fig. 15

AUROC of top 5 teams in CAFA- π. The best-performing model from each team is picked for the top five teams, regardless of whether that model is submitted as model 1. All team methods are in gray while BLAST methods are in red, BLAST computational methods are in blue and expression are in yellow. See Table 3 for description of the baselines