Automated redaction of names in adverse event reports using transformer-based neural networks

Abstract

Background: Automated recognition and redaction of personal identifiers in free text can enable organisations to share data while protecting privacy. This is important in the context of pharmacovigilance since relevant detailed information on the clinical course of events, differential diagnosis, and patient-reported reflections may often only be conveyed in narrative form. The aim of this study is to develop and evaluate a method for automated redaction of person names in English narrative text on adverse event reports. The target domain for this study was case narratives from the United Kingdom's Yellow Card scheme, which collects and monitors information on suspected side effects to medicines and vaccines.

Methods: We finetuned BERT - a transformer-based neural network - for recognising names in case narratives. Training data consisted of newly annotated records from the Yellow Card data and of the i2b2 2014 deidentification challenge. Because the Yellow Card data contained few names, we used predictive models to select narratives for training. Performance was evaluated on a separate set of annotated narratives from the Yellow Card scheme. In-depth review determined whether (parts of) person names missed by the de-identification method could enable re-identification of the individual, and whether de-identification reduced the clinical utility of narratives by collaterally masking relevant information.

Results: Recall on held-out Yellow Card data was 87% (155/179) at a precision of 55% (155/282) and a false-positive rate of 0.05% (127/ 263,451). Considering tokens longer than three characters separately, recall was 94% (102/108) and precision 58% (102/175). For 13 of the 5,042 narratives in Yellow Card test data (71 with person names), the method failed to flag at least one name token. According to in-depth review, the leaked information could enable direct identification for one narrative and indirect identification for two narratives. Clinically relevant information was removed in less than 1% of the 5,042 processed narratives; 97% of the narratives were completely untouched.

Conclusions: Automated redaction of names in free-text narratives of adverse event reports can achieve sufficient recall including shorter tokens like patient initials. In-depth review shows that the rare leaks that occur tend not to compromise patient confidentiality. Precision and false positive rates are acceptable with almost all clinically relevant information retained.

Keywords: Adverse drug reaction reporting systems; Data anonymization; De-identification; Domain adaptation; Medical language processing; Pharmacovigilance.

Fig. 1

Overview of the de-identification method and dataset preparation

Fig. 2

Tokens delimited by the simple alphanumeric tokeniser

Fig. 3

Venn diagram showing the cross-classification of tokens in the Yellow Card test data

Fig. 4

Example narratives where the NAME tokens were correctly flagged by the de-identification method (black background indicating NAME tokens flagged by the model, underline indicating NAME token annotations). N.B. All NAME tokens and personal identifiers are surrogates for similar entities in the original narratives. Drug names and dates have been replaced with placeholders

Fig. 5

Narratives with NAME tokens not flagged by our method (black background indicating NAME tokens flagged by the model, underline indicating NAME token annotations). ‘Ramesh Patel’ was classified as a direct identifier, ‘Kaveson’ was classified as an indirect identifier, and ‘SB’ and ‘deirdre’ were classified as not enabling re-identification in the context of the narratives. N.B. All NAME tokens and personal identifiers are surrogates for similar entities in the original narratives. Drug names and medical facility names have been replaced with placeholders

Fig. 6

Narratives with NON-NAME tokens incorrectly flagged by the method (black background indicating NAME tokens flagged by the model, underline indicating NAME token annotations). For the two narratives to the left, the redacted text was suspected to be clinically relevant; for the two narratives on top, the redacted text was classified as clinically relevant once revealed. N.B. All NAME tokens and personal identifiers are surrogates for similar entities in the original narratives. Drug names have been replaced with a placeholder