Pre-training phenotyping classifiers

Abstract

Recent transformer-based pre-trained language models have become a de facto standard for many text classification tasks. Nevertheless, their utility in the clinical domain, where classification is often performed at encounter or patient level, is still uncertain due to the limitation on the maximum length of input. In this work, we introduce a self-supervised method for pre-training that relies on a masked token objective and is free from the limitation on the maximum input length. We compare the proposed method with supervised pre-training that uses billing codes as a source of supervision. We evaluate the proposed method on one publicly-available and three in-house datasets using the standard evaluation metrics such as the area under the ROC curve and F1 score. We find that, surprisingly, even though self-supervised pre-training performs slightly worse than supervised, it still preserves most of the gains from pre-training.

Keywords: Automatic phenotyping; Natural language processing; Pre-training.

Figure 1:

Masked concept unique identifier (CUI) model. A small number of CUIs from a document are masked and used as prediction targets to train a feed-forward fully-connected neural network. After pre-training, the features computed in the hidden layer can be used as a document representation for training a phenotyping classifier.

Figure 2:

Supervised pre-training. Concept unique identifiers (CUIs) from a document are used to train a feed-forward fully-connected neural network to predict ICD codes. After pre-training, the features computed in the hidden layer can be used as a document representation for training a phenotyping classifier.

Figure 3:

Pre-trained model as a feature extractor. After self-supervised pre-training completes, we save the model and use it to obtain representations of the target data. We then use these representations to train a phenotyping classifier.

Figure 4:

Data pre-processing. All notes associated with a hospital admission (encounter) for a patient are concatenated into a single document and pre-processed with cTAKES to extract UMLS concept unique identifiers (CUIs). For self-supervised pre-training, a small number of CUIs are masked and used as prediction targets. For supervised pre-training, all CUIs are used to predict the billing codes associated with an hospital admission (not shown here).