Assessing the role of a medication-indication resource in the treatment relation extraction from clinical text

Abstract

Objective: To evaluate the contribution of the MEDication Indication (MEDI) resource and SemRep for identifying treatment relations in clinical text.

Materials and methods: We first processed clinical documents with SemRep to extract the Unified Medical Language System (UMLS) concepts and the treatment relations between them. Then, we incorporated MEDI into a simple algorithm that identifies treatment relations between two concepts if they match a medication-indication pair in this resource. For a better coverage, we expanded MEDI using ontology relationships from RxNorm and UMLS Metathesaurus. We also developed two ensemble methods, which combined the predictions of SemRep and the MEDI algorithm. We evaluated our selected methods on two datasets, a Vanderbilt corpus of 6864 discharge summaries and the 2010 Informatics for Integrating Biology and the Bedside (i2b2)/Veteran's Affairs (VA) challenge dataset.

Results: The Vanderbilt dataset included 958 manually annotated treatment relations. A double annotation was performed on 25% of relations with high agreement (Cohen's κ = 0.86). The evaluation consisted of comparing the manual annotated relations with the relations identified by SemRep, the MEDI algorithm, and the two ensemble methods. On the first dataset, the best F1-measure results achieved by the MEDI algorithm and the union of the two resources (78.7 and 80, respectively) were significantly higher than the SemRep results (72.3). On the second dataset, the MEDI algorithm achieved better precision and significantly lower recall values than the best system in the i2b2 challenge. The two systems obtained comparable F1-measure values on the subset of i2b2 relations with both arguments in MEDI.

Conclusions: Both SemRep and MEDI can be used to extract treatment relations from clinical text. Knowledge-based extraction with MEDI outperformed use of SemRep alone, but superior performance was achieved by integrating both systems. The integration of knowledge-based resources such as MEDI into information extraction systems such as SemRep and the i2b2 relation extractors may improve treatment relation extraction from clinical text.

Keywords: MEDI; SemRep; natural language processing; treatment relation extraction.

Figure 1:

The MEDication Indication (MEDI) expansion process.

Figure 2:

The contribution of the medication-indication pairs from MEDI-ALL to each level of medication expansion.

Figure 3:

The diagrams in panels A–D show the connection between the relations identified by the MEDI algorithm and SemRep for each medication expansion level. The diagrams in panels E–H depict the connection between the relation types generated by the two resources. Finally, the diagrams in panels I–L consider the set of relations with their type generated by both resources. MEDI algorithm configuration: MEDI-ALL and the core indication expansion.

Figure 4:

Precision and recall results achieved by the MEDI algorithm for each expansion configuration. The elements of the horizontal axis in each plot correspond to results for each medication expansion level. The results of the algorithm using MEDI-ALL and MEDI-HPS are shown in the first and last two plots from the left, respectively. The results for the core indication expansion are shown in the first and third plot from the left. Finally, the results for the hyponym indication expansion are shown in the second and fourth plot from the left.

Figure 5:

Percentage distributions of medications and indications computed by running the MEDI algorithm over the set of discharge summaries. Each bar plot in red represents the distribution of all indications treated by a specific medication. Similarly, each bar plot in blue shows the distribution of all medications prescribed for a specific indication. In addition to the concept name, each plot title specifies the number of times the corresponding concept was identified in the dataset. For building these distributions, we converted all medication brand names into their corresponding generic names. MEDI algorithm configuration: MEDI-HPS, the Level 3 medication expansion, and the core indication expansion.