Overview of the Cancer Genetics and Pathway Curation tasks of BioNLP Shared Task 2013

Abstract

Background: Since their introduction in 2009, the BioNLP Shared Task events have been instrumental in advancing the development of methods and resources for the automatic extraction of information from the biomedical literature. In this paper, we present the Cancer Genetics (CG) and Pathway Curation (PC) tasks, two event extraction tasks introduced in the BioNLP Shared Task 2013. The CG task focuses on cancer, emphasizing the extraction of physiological and pathological processes at various levels of biological organization, and the PC task targets reactions relevant to the development of biomolecular pathway models, defining its extraction targets on the basis of established pathway representations and ontologies.

Results: Six groups participated in the CG task and two groups in the PC task, together applying a wide range of extraction approaches including both established state-of-the-art systems and newly introduced extraction methods. The best-performing systems achieved F-scores of 55% on the CG task and 53% on the PC task, demonstrating a level of performance comparable to the best results achieved in similar previously proposed tasks.

Conclusions: The results indicate that existing event extraction technology can generalize to meet the novel challenges represented by the CG and PC task settings, suggesting that extraction methods are capable of supporting the construction of knowledge bases on the molecular mechanisms of cancer and the curation of biomolecular pathway models. The CG and PC tasks continue as open challenges for all interested parties, with data, tools and resources available from the shared task homepage.

Figure 1

Illustration of entity annotations. a) Cancer Genetics task b) Pathway Curation task. (Illustrations created with BRAT[59])

Figure 2

Illustration of relation annotations. a) Cancer Genetics task b) Pathway Curation task.

Figure 3

Illustration of event annotations. a) Cancer Genetics task b) Pathway Curation task.

Figure 4

Illustration of the data format. Adapted from [36].

Figure 5

Pathway model reactions and event representations. Illustration of reactions in a pathway model (left), idealized explicit statements annotated with a directly mapped representation (center), and realistic expressions in text with actual event annotation. Figure from [5].

Figure 6

Simple events. Events with single arguments are reliably extracted regardless of factors such as text domain or level or biological organization.

Figure 7

Complex events. Events involving multiple participants, recursive structure, and modifications continue to represent challenges for extraction.