De-identification of patient notes with recurrent neural networks

Abstract

Objective: Patient notes in electronic health records (EHRs) may contain critical information for medical investigations. However, the vast majority of medical investigators can only access de-identified notes, in order to protect the confidentiality of patients. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) defines 18 types of protected health information that needs to be removed to de-identify patient notes. Manual de-identification is impractical given the size of electronic health record databases, the limited number of researchers with access to non-de-identified notes, and the frequent mistakes of human annotators. A reliable automated de-identification system would consequently be of high value.

Materials and methods: We introduce the first de-identification system based on artificial neural networks (ANNs), which requires no handcrafted features or rules, unlike existing systems. We compare the performance of the system with state-of-the-art systems on two datasets: the i2b2 2014 de-identification challenge dataset, which is the largest publicly available de-identification dataset, and the MIMIC de-identification dataset, which we assembled and is twice as large as the i2b2 2014 dataset.

Results: Our ANN model outperforms the state-of-the-art systems. It yields an F1-score of 97.85 on the i2b2 2014 dataset, with a recall of 97.38 and a precision of 98.32, and an F1-score of 99.23 on the MIMIC de-identification dataset, with a recall of 99.25 and a precision of 99.21.

Conclusion: Our findings support the use of ANNs for de-identification of patient notes, as they show better performance than previously published systems while requiring no manual feature engineering.

Keywords: de-identification; medical language processing; neural networks.

Figure 1.

Architecture of the artificial neural network (ANN) model. (RNN, recurrent neural network.) The type of RNN used in this model is long short-term memory (LSTM). $n$ is the number of tokens, and $x_i$ is the $i^{th}$ token. ${\mathcal{V}}_T$ is the mapping from tokens to token embeddings. $\ell (i)$ is the number of characters and $x_{i,j}$ is the $j^{th}$ character in the $i^{th}$ token. ${\mathcal{V}}_C$ is the mapping from characters to character embeddings. ${\mathrm{e}}_i$ is the character-enhanced token embeddings of the $i^{\mathrm{t}\mathrm{h}}$ token. ${\overleftrightarrow{\mathbf{d}}}_i$ is the output of the LSTM of the label prediction layer, ${\mathbf{a}}_i$ is the probability vector over labels, $y_i$ is the predicted label of the i^th token.

Figure 2.

Binary token-based F1-scores for each PHI category. The evaluation is based on PHI types that are defined by HIPAA as well as additional types specific to each dataset. Each PHI category and the corresponding types are defined in Table 1. The “All” category refers to the F1-score micro-averaged over all PHI categories. The PROFESSION category exists only in the i2b2 dataset and was plotted separately to avoid distorting the y-axis. For the same reason, the AGE category in MIMIC was drawn separately.

Figure 3.

Impact of the training set size on the binary HIPAA token-based F1-scores on the MIMIC dataset. The 100% training set size refers to using all of the dataset minus the test set, which amounts to 2 046 488 tokens and 42 531 PHI instances. As expected, both CRF and ANN models benefit from having more training samples.

Figure 4.

Impact of the number of labeled PHI instances in the training set on the model’s performance for each PHI type in the i2b2 dataset. Figure (A) presents all PHI types, and Figure (B) focuses on the most commonly occurring PHI types. Having more PHI instances in the training set helps increase F1-score, but some PHI types are harder to detect than others.

Figure 5.

Ablation test performance based on binary HIPAA token-based evaluation. ANN is the model based on artificial neural networks. “$-$ seq opt” is the ANN model without the label sequence optimization layer. “$-$ pre-train” is the ANN model where token embeddings are initialized with random values instead of pre-trained embeddings. “$-$ token emb” is the ANN model using only character-based token embeddings, without token embeddings. “$-$ character emb” is the ANN model using only token embeddings, without character-based token embeddings.