Deciphering clinical abbreviations with a privacy protecting machine learning system

Abstract

Physicians write clinical notes with abbreviations and shorthand that are difficult to decipher. Abbreviations can be clinical jargon (writing "HIT" for "heparin induced thrombocytopenia"), ambiguous terms that require expertise to disambiguate (using "MS" for "multiple sclerosis" or "mental status"), or domain-specific vernacular ("cb" for "complicated by"). Here we train machine learning models on public web data to decode such text by replacing abbreviations with their meanings. We report a single translation model that simultaneously detects and expands thousands of abbreviations in real clinical notes with accuracies ranging from 92.1%-97.1% on multiple external test datasets. The model equals or exceeds the performance of board-certified physicians (97.6% vs 88.7% total accuracy). Our results demonstrate a general method to contextually decipher abbreviations and shorthand that is built without any privacy-compromising data.

Fig. 1. Overview of task formulation and comparison against traditional approaches.

A Task formulation: The input to our model is a string that may or may not contain medical abbreviations. We trained a model to output a corresponding string in which all abbreviations are simultaneously detected and expanded. If the input string does not contain an abbreviation, the model will output the original string. B Traditional vs. our approach: The traditional approach for medical abbreviation disambiguation divides the task into separate steps. First, a separate component identifies which tokens of an input string are abbreviations. Second, each abbreviation’s surrounding context is processed by an abbreviation-specific model that outputs scores associated with a closed set of possible expansions of the abbreviation. This not only requires training and deploying thousands of models, but each model also cannot benefit from the inductive bias of the other models, especially as it relates to any other abbreviations that may appear in the surrounding context of a given abbreviation. Finally, each expansion must be inserted back into the original snippet, which may require additional post-processing to maintain grammatical correctness. In our approach, a single end-to-end process performs all of these steps in parallel: the detection of abbreviations, the expansion of all abbreviations such that their likelihood is maximized in a mutually consistent manner, and the generation of a grammatically consistent “decoded” piece of text.

Fig. 2. Overview of web-scale reverse substitution.

Web scale reverse substitution: Large language models are often pre-trained on public web data to carry out general self-supervised tasks. To further train (i.e. fine-tune) a model to decipher clinical abbreviations, we use the following procedure to generate an additional dataset as shown conceptually in this panel. We take public web pages and identify words or phrases that have corresponding abbreviations (the colored boxes on the left-hand side) in the dictionary released along with this manuscript. We substitute the abbreviation (the colored dots) to generate input text to train our model for the decoding task. If a word/phrase has more than one abbreviation (e.g. “atrial fibrillation” could be “af’ or “afib”), we randomly pick one. Given the size of the web-corpus and the imbalance of expansions (e.g. “patient” is found orders of magnitude more often than “posterior tibialis”), a simple “find and replace” is problematic because it creates a large, imbalanced dataset. Instead, our algorithm, which we call web-scale reverse substitution, downsamples frequent expansions to derive a more balanced dataset from a web-sized corpus of thousands of expansions.

Fig. 3. Effect of model size and inference type on performance.

Model size and inference type influence key model metrics on the synthetic test set. Each point reflects a T5 model with identical pre-training, MC-WSRS fine-tuning, and evaluation on a synthetic dataset of medical snippets. Detection recall decreases as the model size increases. However, performance is substantially and statistically improved with inference chaining techniques such as iterative inference and elicitive inference. The inference types and model sizes do not significantly affect detection precision (percentage of the text identified as abbreviations that are actually abbreviations), and model size improves expansion accuracy (percentage of expansions with clinically equivalent meanings). Total accuracy, which we define as detection recall multiplied by expansion accuracy, is highest for the model with the most parameters with elicitive inference. n = 400 bootstrap samples of the 302 synthetic snippets, each of which contains a different collection of abbreviations. Point estimates from the original sample and 95% confidence intervals were calculated using reporting the 2.5 and 97.5 percentile values for each metric across the samples.