Functional evaluation of out-of-the-box text-mining tools for data-mining tasks

Abstract

Objective: The trade-off between the speed and simplicity of dictionary-based term recognition and the richer linguistic information provided by more advanced natural language processing (NLP) is an area of active discussion in clinical informatics. In this paper, we quantify this trade-off among text processing systems that make different trade-offs between speed and linguistic understanding. We tested both types of systems in three clinical research tasks: phase IV safety profiling of a drug, learning adverse drug-drug interactions, and learning used-to-treat relationships between drugs and indications.

Materials: We first benchmarked the accuracy of the NCBO Annotator and REVEAL in a manually annotated, publically available dataset from the 2008 i2b2 Obesity Challenge. We then applied the NCBO Annotator and REVEAL to 9 million clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE) and used the resulting data for three research tasks.

Results: There is no significant difference between using the NCBO Annotator and REVEAL in the results of the three research tasks when using large datasets. In one subtask, REVEAL achieved higher sensitivity with smaller datasets.

Conclusions: For a variety of tasks, employing simple term recognition methods instead of advanced NLP methods results in little or no impact on accuracy when using large datasets. Simpler dictionary-based methods have the advantage of scaling well to very large datasets. Promoting the use of simple, dictionary-based methods for population level analyses can advance adoption of NLP in practice.

Keywords: electronic health records; natural language processing; text mining.

Figure 1:

Our investigation has three parts. (A) First, we benchmark the accuracy of the NCBO Annotator-based workflow, REVEAL, and cTAKES on the task of finding mentions of co-morbidities in the 2008 i2b2 Obesity Challenge dataset (details in figure 2). (B) Second, we evaluate the trade-off of using annotations, and the resulting patient-feature matrix, from 9 million clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE) generated using the NCBO Annotator-based workflow and the REVEAL natural language processing (NLP) system. The three research tasks are: detection of used-to-treat relationships between drugs and indications, detection of drug–drug interactions, and profiling the safety of cilostazol use in patients with peripheral artery disease (PAD). Each of these evaluations is based on previously published work; the only source of variation is the annotations used as input to the published methods. The patient-feature matrix is described in detail in online supplemental materials S2. We did not run cTAKES on the 9 million clinical notes from STRIDE because it would have required over a year to complete given our computational resources. (C) Finally, we explore the impact of dataset size on the task of detecting the used-to-treat relationship using increasingly smaller subsets of the data (details in figure 6).

Figure 2:

The precision and recall of the NCBO Annotator, REVEAL, and cTAKES in the 2008 i2b2 dataset is plotted here for each indication. There is considerable variation in recall across both systems and indications, but generally indications that are hard to detect are hard to detect for all systems (eg, Gallstones, labeled here as GS). There is no universally best system across all indications with respect to either precision or recall. ASA, asthma; CAD, coronary artery disease; CHF, congestive heart failure; DM, diabetes; DEPR, depression; GS, gallstones; GERD, gastro-esophageal reflux disease; GT, gout; HCL, hypercholesterolemia; HTN, hypertension; HTG, hypertriglyceridemia; OA, osteoarthritis; OBS, obesity; OSA, obstructive sleep apnea; PVD, peripheral vascular disease.

Figure 3:

Profile of adverse events in peripheral artery disease patients with and without exposure to cilostazol. The plot shows ORs and 95% CIs calculated using annotations from the Stanford Translational Research Integrated Database Environment (STRIDE) using the NCBO Annotator-based workflow and REVEAL. There is no change in the conclusions of this analysis depending on the text processing system being used. Note that REVEAL did not find any instances of ‘sudden cardiac death’ in the data; for this event, we set the OR to 1. MACE, major adverse cardiac event.

Figure 4:

Detection of adverse drug–drug interactions. The analysis of Iyer et al. was carried out using either the NCBO Annotator-based workflow or REVEAL to process clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE). (A) There is no significant difference in the receiver-operator characteristic (ROC) curves (p = 0.275 by DeLong's test) for the two systems. (B) Area under the ROC curves (AUC) for each of nine adverse events separately. There is no significant difference in performance for any adverse event (p>0.05).

Figure 5:

Detecting used-to-treat relationships. We carried out the analysis described in Jung et al. using either the NCBO Annotator or REVEAL to annotate clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE). There is no significant difference in performance between classifiers trained and tested using features derived from either (p = 0.29 by McNemar's test). PPV, positive predictive value.

Figure 6:

Learning curves for the used-to-treat task. We sampled random subsets of patients and used the associated notes to generate features based on the annotations of those notes by either the NCBO Annotator or REVEAL. This was repeated 10 times for each fraction of the full Stanford Translational Research Integrated Database Environment (STRIDE) dataset. The mean performance metric across the 10 runs is plotted, along with the SEM. REVEAL has higher sensitivity in smaller datasets, and generally has higher precision/positive predictive value (PPV).