Validation of Artificial Intelligence to Support the Automatic Coding of Patient Adverse Drug Reaction Reports, Using Nationwide Pharmacovigilance Data

Abstract

Introduction: Adverse drug reaction reports are usually manually assessed by pharmacovigilance experts to detect safety signals associated with drugs. With the recent extension of reporting to patients and the emergence of mass media-related sanitary crises, adverse drug reaction reports currently frequently overwhelm pharmacovigilance networks. Artificial intelligence could help support the work of pharmacovigilance experts during such crises, by automatically coding reports, allowing them to prioritise or accelerate their manual assessment. After a previous study showing first results, we developed and compared state-of-the-art machine learning models using a larger nationwide dataset, aiming to automatically pre-code patients' adverse drug reaction reports.

Objectives: We aimed to determine the best artificial intelligence model identifying adverse drug reactions and assessing seriousness in patients reports from the French national pharmacovigilance web portal.

Methods: Reports coded by 27 Pharmacovigilance Centres between March 2017 and December 2020 were selected (n = 11,633). For each report, the Portable Document Format form containing free-text information filled by the patient, and the corresponding encodings of adverse event symptoms (in Medical Dictionary for Regulatory Activities Preferred Terms) and seriousness were obtained. This encoding by experts was used as the reference to train and evaluate models, which contained input data processing and machine-learning natural language processing to learn and predict encodings. We developed and compared different approaches for data processing and classifiers. Performance was evaluated using receiver operating characteristic area under the curve (AUC), F-measure, sensitivity, specificity and positive predictive value. We used data from 26 Pharmacovigilance Centres for training and internal validation. External validation was performed using data from the remaining Pharmacovigilance Centres during the same period.

Results: Internal validation: for adverse drug reaction identification, Term Frequency-Inverse Document Frequency (TF-IDF) + Light Gradient Boosted Machine (LGBM) achieved an AUC of 0.97 and an F-measure of 0.80. The Cross-lingual Language Model (XLM) [transformer] obtained an AUC of 0.97 and an F-measure of 0.78. For seriousness assessment, FastText + LGBM achieved an AUC of 0.85 and an F-measure of 0.63. CamemBERT (transformer) + Light Gradient Boosted Machine obtained an AUC of 0.84 and an F-measure of 0.63. External validation for both adverse drug reaction identification and seriousness assessment tasks yielded consistent and robust results.

Conclusions: Our artificial intelligence models showed promising performance to automatically code patient adverse drug reaction reports, with very similar results across approaches. Our system has been deployed by national health authorities in France since January 2021 to facilitate pharmacovigilance of COVID-19 vaccines. Further studies will be needed to validate the performance of the tool in real-life settings.

Fig. 1

Summary of the methodological differences between tasks. ADRs adverse drug reactions, ER emergency room, LGBM Light Gradient Boosted Machine, PT Preferred Term, XLM Cross-lingual Language Model

Fig. 2

Flowchart of the selection of datasets. ADR adverse drug reaction, ANSM French National Agency for the Safety of Medicines and Health Products, CRPV Regional Pharmacovigilance Centre

Fig. 3

Repartition of the most frequently reported drugs and adverse drug reactions (ADRs) coded in Medical Dictionary for Regulatory Activities Preferred Terms (PTs) across complete datasets. A Mosaic plot of the most frequently reported drugs and B mosaic plot of the most frequently reported PTs

Fig. 4

Adverse drug reaction identification: receiver operating characteristic (ROC) and precision-recall (PR) curves. A ROC curve of internal validation, B PR curve of internal validation, C ROC curve of external validation and D PR curve of external validation

Fig. 5

Seriousness assessment: receiver operating characteristic (ROC) and precision-recall (PR) curves. A ROC curve of internal validation, B PR curve of internal validation, with dataset seriousness prevalence at y = 0.24, C ROC curve of external validation and D PR curve of external validation, with dataset seriousness prevalence at y = 0.21