Extracting similar terms from multiple EMR-based semantic embeddings to support chart reviews

Abstract

Objective: Word embeddings project semantically similar terms into nearby points in a vector space. When trained on clinical text, these embeddings can be leveraged to improve keyword search and text highlighting. In this paper, we present methods to refine the selection process of similar terms from multiple EMR-based word embeddings, and evaluate their performance quantitatively and qualitatively across multiple chart review tasks.

Materials and methods: Word embeddings were trained on each clinical note type in an EMR. These embeddings were then combined, weighted, and truncated to select a refined set of similar terms to be used in keyword search and text highlighting. To evaluate their quality, we measured the similar terms' information retrieval (IR) performance using precision-at-K (P@5, P@10). Additionally a user study evaluated users' search term preferences, while a timing study measured the time to answer a question from a clinical chart.

Results: The refined terms outperformed the baseline method's information retrieval performance (e.g., increasing the average P@5 from 0.48 to 0.60). Additionally, the refined terms were preferred by most users, and reduced the average time to answer a question.

Conclusions: Clinical information can be more quickly retrieved and synthesized when using semantically similar term from multiple embeddings.

Keywords: Clinical similar terms; Electronic medical records (EMR); Highlighting; Query expansion; Search engines; Semantic embeddings.

Figure 1.

Similar terms of “cancer” from the “Clinic Note” EMR-subset embedding broken down by intrasubset similarity, inter-subsets similarity, and harmonic similarity. The harmonic similarity is used for ranking terms.

Figure 2.

Example of expanded document quality analysis for “epilepsy.” The proportion of high similarity terms (i.e. terms that have similarities larger than 0.60 while 1.0 is the maximum value) decreases with similar term expansion.

Figure 3.

Example of similarity cutoff computation. Since all terms have similarities larger than 0.40, the y-axis starts from 0.3. Similarity cutoff is at the “elbow” of the similarity curve (arrow).

Figure 4.

Screenshot of the preference survey. An introduction is provided, followed by 14 questions that ask the participant to choose the best list to expand a keyword. List orders were randomized to hide source methods.