Ensemble method-based extraction of medication and related information from clinical texts

Abstract

Objective: Accurate and complete information about medications and related information is crucial for effective clinical decision support and precise health care. Recognition and reduction of adverse drug events is also central to effective patient care. The goal of this research is the development of a natural language processing (NLP) system to automatically extract medication and adverse drug event information from electronic health records. This effort was part of the 2018 n2c2 shared task on adverse drug events and medication extraction.

Materials and methods: The new NLP system implements a stacked generalization based on a search-based structured prediction algorithm for concept extraction. We trained 4 sequential classifiers using a variety of structured learning algorithms. To enhance accuracy, we created a stacked ensemble consisting of these concept extraction models trained on the shared task training data. We implemented a support vector machine model to identify related concepts.

Results: Experiments with the official test set showed that our stacked ensemble achieved an F1 score of 92.66%. The relation extraction model with given concepts reached a 93.59% F1 score. Our end-to-end system yielded overall micro-averaged recall, precision, and F1 score of 92.52%, 81.88% and 86.88%, respectively. Our NLP system for adverse drug events and medication extraction ranked within the top 5 of teams participating in the challenge.

Conclusion: This study demonstrated that a stacked ensemble with a search-based structured prediction algorithm achieved good performance by effectively integrating the output of individual classifiers and could provide a valid solution for other clinical concept extraction tasks.

Keywords: drug-related side effects and adverse reactions [MeSH C25.100]; medical informatics [L01.313.500]; natural language processing (NLP) [L01.224.050.375.580]; neural networks [L01.224.050.375.605].

Figure 1.

Sample text with (A) concept annotations and (B) examples of each concept category. ADE: adverse drug event.

Figure 2.

Concept and relation category counts distribution. ADE: adverse drug event.

Figure 3.

System architecture of the end-to-end system. CRF: conditional random fields; EHR: electronic health record; RNN: recurrent neural network; SEARN: search-based structured prediction; SVM: support vector machine.

Figure 4.

F1 score of each external resource (lenient matching). i2b2: Informatics for Integrating Biology and the Beside; CADEC: CSIRO Adverse Drug Events Corpus.