Community challenges in biomedical text mining over 10 years: success, failure and the future

Abstract

One effective way to improve the state of the art is through competitions. Following the success of the Critical Assessment of protein Structure Prediction (CASP) in bioinformatics research, a number of challenge evaluations have been organized by the text-mining research community to assess and advance natural language processing (NLP) research for biomedicine. In this article, we review the different community challenge evaluations held from 2002 to 2014 and their respective tasks. Furthermore, we examine these challenge tasks through their targeted problems in NLP research and biomedical applications, respectively. Next, we describe the general workflow of organizing a Biomedical NLP (BioNLP) challenge and involved stakeholders (task organizers, task data producers, task participants and end users). Finally, we summarize the impact and contributions by taking into account different BioNLP challenges as a whole, followed by a discussion of their limitations and difficulties. We conclude with future trends in BioNLP challenge evaluations.

Keywords: BioNLP challenges; BioNLP shared tasks; biomedical natural language processing (BioNLP); critical assessment; text mining.

Figure 1

Data channeling based on NLP technology and the application of the channeled data. NLP technology (e.g. Document Retrieval or Information Extraction) helps alleviate scientists in biology and life science from significant efforts of manual searching/researching for text snippets of interest by narrowing down the search space. The topics of BioNLP challenge tasks with the focus of the NLP technology are exemplified. For instance, topics associated with Information Extraction in BioNLP include, but not limit to, finding drug–drug interactions, protein–protein interactions, gene relations, clinical temporal relations and references into gene functions. The channeled/text-mined data, on the other hand, can be further used to curate databases, construct ontologies, build semantic networks or interactive systems and so on.

Figure 2

BioNLP challenges in chronological order. Challenges are shown in bold white font, whereas their specific task focus is shown in italic black font following the task/track short-hands in Table 1.

Figure 3

Challenges’ subtasks/tracks organized based on NLP perspectives. A colour version of this figure is available at BIB online: http://bib.oxfordjournals.org.

Figure 4

Different biological and clinical problems targeted by BioNLP challenges. Challenge subtasks are coded in the same colors as in Figure 2 (e.g. BioNLP-ST tasks are green marked). A colour version of this figure is available at BIB online: http://bib.oxfordjournals.org.

Figure 5

The typical workflow of organizing a shared task.

Figure 6

Impact/contributions from BioNLP challenges.