BERT based natural language processing for triage of adverse drug reaction reports shows close to human-level performance

Abstract

Post-marketing reports of suspected adverse drug reactions are important for establishing the safety profile of a medicinal product. However, a high influx of reports poses a challenge for regulatory authorities as a delay in identification of previously unknown adverse drug reactions can potentially be harmful to patients. In this study, we use natural language processing (NLP) to predict whether a report is of serious nature based solely on the free-text fields and adverse event terms in the report, potentially allowing reports mislabelled at time of reporting to be detected and prioritized for assessment. We consider four different NLP models at various levels of complexity, bootstrap their train-validation data split to eliminate random effects in the performance estimates and conduct prospective testing to avoid the risk of data leakage. Using a Swedish BERT based language model, continued language pre-training and final classification training, we achieve close to human-level performance in this task. Model architectures based on less complex technical foundation such as bag-of-words approaches and LSTM neural networks trained with random initiation of weights appear to perform less well, likely due to the lack of robustness that a base of general language training provides.

Fig 1. Current dataset quarterly distribution with information on seriousness.

Data as available at time of database lock.

Fig 2. Overview of the data flow and model architectures investigated.

Fig 3. Density plots of F1 scores for all models (left) and F1 difference between models using BERT and AER-BERT (right).

Fig 4. Receiver operating characteristics (left) and Precision-Recall curve (right) for the four model architectures on the prospective test set.

Fig 5. Results from hold-out sample with two human assessors and the four models.

We show predictions on the serious (left) and non-serious reports (right) separately for each class, where the leftmost column in the prediction heatmap always corresponds to the database annotation.