The contribution of co-reference resolution to supervised relation detection between bacteria and biotopes entities

Abstract

Background: The acquisition of knowledge about relations between bacteria and their locations (habitats and geographical locations) in short texts about bacteria, as defined in the BioNLP-ST 2013 Bacteria Biotope task, depends on the detection of co-reference links between mentions of entities of each of these three types. To our knowledge, no participant in this task has investigated this aspect of the situation. The present work specifically addresses issues raised by this situation: (i) how to detect these co-reference links and associated co-reference chains; (ii) how to use them to prepare positive and negative examples to train a supervised system for the detection of relations between entity mentions; (iii) what context around which entity mentions contributes to relation detection when co-reference chains are provided.

Results: We present experiments and results obtained both with gold entity mentions (task 2 of BioNLP-ST 2013) and with automatically detected entity mentions (end-to-end system, in task 3 of BioNLP-ST 2013). Our supervised mention detection system uses a linear chain Conditional Random Fields classifier, and our relation detection system relies on a Logistic Regression (aka Maximum Entropy) classifier. They use a set of morphological, morphosyntactic and semantic features. To minimize false inferences, co-reference resolution applies a set of heuristic rules designed to optimize precision. They take into account the types of the detected entity mentions, and take advantage of the didactic nature of the texts of the corpus, where a large proportion of bacteria naming is fairly explicit (although natural referring expressions such as "the bacteria" are common). The resulting system achieved a 0.495 F-measure on the official test set when taking as input the gold entity mentions, and a 0.351 F-measure when taking as input entity mentions predicted by our CRF system, both of which are above the best BioNLP-ST 2013 participant system.

Conclusions: We show that co-reference resolution substantially improves over a baseline system which does not use co-reference information: about 3.5 F-measure points on the test corpus for the end-to-end system (5.5 points on the development corpus) and 7 F-measure points on both development and test corpora when gold mentions are used. While this outperforms the best published system on the BioNLP-ST 2013 Bacteria Biotope dataset, we consider that it provides mostly a stronger baseline from which more work can be started. We also emphasize the importance and difficulty of designing a comprehensive gold standard co-reference annotation, which we explain is a key point to further progress on the task.

Figure 1

Global view of the process.

Figure 2

Annotated corpus excerpt. Excerpt from the annotated corpus (BTID-10087 file, training corpus) using the BRAT Rapid Annotation Tool.

Figure 3

Co-reference between similar Bacteria mentions. Co-reference relation between graphically similar Bacteria entity mentions Campylobacter coli and C. coli: these mentions and this instance of co-reference relation were given in the gold standard annotations provided with the training corpus of Task 2. This co-reference relation is instrumental in the detection of the Localization relations between Campylobacter coli and its habitats pigs, birds, and surface water. Ellipsis (...) shows skipped material.

Figure 4

Co-reference through definite noun phrase anaphora to a Bacteria-type mention. Co-reference relation through definite noun phrase anaphora to a Bacteria-type mention: The organism refers to Bacteria-type mention Yersinia pestis. Since The organism is not the name of a bacterium, it was not annotated in the gold standard annotations provided with the training corpus of Task 2, nor was the corresponding co-reference relation. Since sentence The organism ... infection asserts the locations where The organism can be found, it is important to capture the co-reference between This organism and Yersinia pestis to detect the Localization relations between this bacterium and its habitats. Ellipsis (...) shows skipped material.

Figure 5

Impact on relation detection of the maximum distance (in sentences) between entity mentions when training (training corpus). Variation of performance with the maximum distance s between entity mentions at training time.

Figure 6

Impact on relation detection of the maximum distance (in sentences) between entity mentions when decoding (development corpus). Variation of performance with the maximum distance s between entity mentions at decoding time.

Figure 7

Impact on relation detection of the size of the context surrounding each mention. Variation of performance with the size n of the left and right contexts used to collect features for each entity mention on the development corpus.