Automated vocabulary discovery for geo-parsing online epidemic intelligence

Abstract

Background: Automated surveillance of the Internet provides a timely and sensitive method for alerting on global emerging infectious disease threats. HealthMap is part of a new generation of online systems designed to monitor and visualize, on a real-time basis, disease outbreak alerts as reported by online news media and public health sources. HealthMap is of specific interest for national and international public health organizations and international travelers. A particular task that makes such a surveillance useful is the automated discovery of the geographic references contained in the retrieved outbreak alerts. This task is sometimes referred to as "geo-parsing". A typical approach to geo-parsing would demand an expensive training corpus of alerts manually tagged by a human.

Results: Given that human readers perform this kind of task by using both their lexical and contextual knowledge, we developed an approach which relies on a relatively small expert-built gazetteer, thus limiting the need of human input, but focuses on learning the context in which geographic references appear. We show in a set of experiments, that this approach exhibits a substantial capacity to discover geographic locations outside of its initial lexicon.

Conclusion: The results of this analysis provide a framework for future automated global surveillance efforts that reduce manual input and improve timeliness of reporting.

Figure 1

Example of training data. An illustration of the word-withholding strategy.

Figure 2

Words sparse representation. Each word in the dictionary, its part-of-speech in the text and its capitalization status are associated with 3 indexes in the representation. If a word is out of the dictionary, it will only be represented by its part-of-speech and capitalization status.

Figure 3

Percentage of artificial out-of-vocabulary words. Percentage of "hidden" words when reducing the dictionary size according to the minimum frequency thresholds λ. The first bar at each λ value shows the number of out-of-vocabulary words among the words of the corpus, and the second bar shows the number of location words outside the vocabulary among the words tagged as location references using the HealthMap gazetteer. Between brackets, the dictionary size corresponding to λ is reported.

Figure 4

Percentage of natural out-of-vocabulary words. Percentage of unique words from a separated evaluation set (500 alerts, 11,184 unique words) that are inside or outside of the training set extracted dictionary, for training sets T₀ (1,000 alerts), T₁ (2,500 alerts) and T₂ (5,000 alerts). The percentage of location words is computed with respect to the locations found by the commercial geo-parser (see sect.).

Figure 5

Evaluation with respect to HealthMap gazetteer tags. F₁ score with respect to HealthMap gazetteer tags for several values of λ (red plain line). F₁ scores among words with visible lexical index (green dashed line) and among words with hidden dictionary's index (blue pointed-dashed line)

Figure 6

Evaluation with respect to MetaCarta tags. Precision, recall and F₁-score with respect to MetaCarta labels for increasing dictionary cut-offs according to the λ threshold. Performances of models trained on T₀ (1,000 alerts), T₁ (2,500 alerts), T₂ (5,000 alerts) and T₁ with location and disease targets.

Figure 7

Illustration of the neural network architecture. An illustration of geo-parsing neural network with a typical input.