Natural language processing improves reliable identification of COVID-19 compared to diagnostic codes alone

Abstract

Observational COVID-19 studies often rely on diagnostic codes, but their accuracy and potential for differential misclassification across patient subgroups are unclear. In this proof of concept study, we examined age, race, and ethnicity as predictors of differential misclassification by comparing the classification accuracy of diagnostic codes to classifiers based on natural language processing (NLP) of clinical notes. We assessed differential misclassification in two primary care-based samples from the American Family Cohort: first, a cohort of 5000 patients with COVID-19 status assessed by physicians based on notes; and second, 21 659 patients (of 1 560 564) who received COVID-specific antivirals. Using annotated note data, we trained and tested three NLP classifiers (tree-based, recurrent neural network, and transformer-based). Approximately 63% of likely COVID-19 patients in the two samples had a documented ICD-10 code for COVID-19. Sensitivity was highest among younger patients (68.6% for <18 years versus 60.6% for those 75+), and for Hispanic patients (68.0% vs 58.5% for Black/African American patients). The tree-based classifier had the highest area under the ROC curve (0.92), although it was less accurate among older patients. NLP performance drastically worsened predicting data collected post-training. While NLP may improve cohort identification, frequent retraining is likely needed to capture changing documentation.

Keywords: COVID-19; cohort identification; natural language processing; sample sizes.

Figure 1:

Receiver operating characteristic curve for the three NLP classifiers. AUC = Area Under Curve, RNN = Recurrent Neural Network.

Figure 2:

ROC curves of the XGBoost classifier based on training datasets of different sizes. Training data of 3000 samples or fewer uses 100 bootstrap samples to determine the mean ROC curve (confidence intervals omitted for clarity, but available in text).