Crowdsourcing biocuration: The Community Assessment of Community Annotation with Ontologies (CACAO)

Abstract

Experimental data about gene functions curated from the primary literature have enormous value for research scientists in understanding biology. Using the Gene Ontology (GO), manual curation by experts has provided an important resource for studying gene function, especially within model organisms. Unprecedented expansion of the scientific literature and validation of the predicted proteins have increased both data value and the challenges of keeping pace. Capturing literature-based functional annotations is limited by the ability of biocurators to handle the massive and rapidly growing scientific literature. Within the community-oriented wiki framework for GO annotation called the Gene Ontology Normal Usage Tracking System (GONUTS), we describe an approach to expand biocuration through crowdsourcing with undergraduates. This multiplies the number of high-quality annotations in international databases, enriches our coverage of the literature on normal gene function, and pushes the field in new directions. From an intercollegiate competition judged by experienced biocurators, Community Assessment of Community Annotation with Ontologies (CACAO), we have contributed nearly 5,000 literature-based annotations. Many of those annotations are to organisms not currently well-represented within GO. Over a 10-year history, our community contributors have spurred changes to the ontology not traditionally covered by professional biocurators. The CACAO principle of relying on community members to participate in and shape the future of biocuration in GO is a powerful and scalable model used to promote the scientific enterprise. It also provides undergraduate students with a unique and enriching introduction to critical reading of primary literature and acquisition of marketable skills.

Fig 1. CACAO competitors contribute a large number of GO annotations.

Overall CACAO contributions are summarized in the context of the workflow for quality control and submission to the GO Consortium. CACAO users consume the primary literature, collect information about normal gene functions from the paper study subjects, and capture the evidence and conclusions using the GO. Those annotations are reviewed by trained judges and marked as unacceptable (red X), requiring changes (yellow!, or purple? flagged for further review), or acceptable (green check, or blue check after correction) within the GONUTS framework. Competitors challenge entries and engage in peer review until an annotation is corrected or marked unacceptable. Fully vetted annotations are deposited into the public GO database maintained by professional biocurators and used by scientists worldwide. As required, CACAO-submitted annotations will be updated to reflect rearrangements and changes in GO. CACAO, Community Assessment of Community Annotation with Ontologies; GO, Gene Ontology; GONUTS, Gene Ontology Normal Usage Tracking System.

Fig 2. The GO annotations contributed by CACAO users are diverse and specific.

(A) Proteins annotated by CACAO users are depicted by species domain. The organisms most highly represented in each domain are displayed on the outer ring of the chart divided by the following rank: phylum for eukaryotes and archaea, order for bacteria, and family for viruses. The number of GO annotations in each category is indicated in brackets. (B) The distribution of GO terms used for CACAO annotations are graphed by aspect within the ontology. The top 3 terms within each aspect are labeled on the outer ring. For clarity, “activity” was dropped from each function term, and the process terms were abbreviated from “positive/negative regulation of transcription, DNA-templated” to “transcript. reg., positive or negative.” The number of GO annotations for each term is indicated in brackets. (C) The descendant counts, corresponding to depth within the ontology, for CACAO annotations (n = 4,913) and all other manual GO annotations in UniProt through 2019 (n = 255,958) are graphed. Significant differences measured by the Mann–Whitney test with p<0.001 are marked with an *. CACAO, Community Assessment of Community Annotation with Ontologies; GO, Gene Ontology.