Multi-label classification of symptom terms from free-text bilingual adverse drug reaction reports using natural language processing

Abstract

Allergic reactions to medication range from mild to severe or even life-threatening. Proper documentation of patient allergy information is critical for safe prescription, avoiding drug interactions, and reducing healthcare costs. Allergy information is regularly obtained during the medical interview, but is often poorly documented in electronic health records (EHRs). While many EHRs allow for structured adverse drug reaction (ADR) reporting, a free-text entry is still common. The resulting information is neither interoperable nor easily reusable for other applications, such as clinical decision support systems and prescription alerts. Current approaches require pharmacists to review and code ADRs documented by healthcare professionals. Recently, the effectiveness of machine algorithms in natural language processing (NLP) has been widely demonstrated. Our study aims to develop and evaluate different NLP algorithms that can encode unstructured ADRs stored in EHRs into institutional symptom terms. Our dataset consists of 79,712 pharmacist-reviewed drug allergy records. We evaluated three NLP techniques: Naive Bayes-Support Vector Machine (NB-SVM), Universal Language Model Fine-tuning (ULMFiT), and Bidirectional Encoder Representations from Transformers (BERT). We tested different general-domain pre-trained BERT models, including mBERT, XLM-RoBERTa, and WanchanBERTa, as well as our domain-specific AllergyRoBERTa, which was pre-trained from scratch on our corpus. Overall, BERT models had the highest performance. NB-SVM outperformed ULMFiT and BERT for several symptom terms that are not frequently coded. The ensemble model achieved an exact match ratio of 95.33%, a F1 score of 98.88%, and a mean average precision of 97.07% for the 36 most frequently coded symptom terms. The model was then further developed into a symptom term suggestion system and achieved a Krippendorff's alpha agreement coefficient of 0.7081 in prospective testing with pharmacists. Some degree of automation could both accelerate the availability of allergy information and reduce the efforts for human coding.

Fig 1. Flowchart outlines the steps involved in our study.

Drug allergy records in the Songklanagarind Hospital’s EHR from October 2001 to July 2020 were extracted for developing algorithms. Only the records that had been reviewed by pharmacists were considered. We then trained NB-SVM, ULMFiT, and BERT-based models to map the unstructured allergy description to the institutional symptom terms. The ensemble model was then used for evaluation with pharmacists via a web application in a simulated EHR environment.

Fig 2. GUI for document a new drug allergy record.

GUI for documenting a drug allergy record in the Songklanagarind Hospital’s EHR. The user interface was originally in Thai and was translated to English for ease of the reader.

Fig 3. Diagrams outline the steps involved in our methods.

(A) Data preparation included word segmentation and tokenization, with each algorithm using different method. (B) NB-SVM involved training multiple pipelines for Naive Bayes feature extraction and SVM classification. (C) ULMFiT involved fine-tuning the pre-trained LM with our target-domain allergy corpus and fine-tuning a classifier for our multi-label classification task. (D) BERT involves fine-tuning a classifier for our multi-label classification task. This study evaluated three pre-trained general-domain BERT models and one target-domain BERT model pre-trained on our allergy corpus.