Deep learning uncertainty quantification for clinical text classification

Abstract

Introduction: Machine learning algorithms are expected to work side-by-side with humans in decision-making pipelines. Thus, the ability of classifiers to make reliable decisions is of paramount importance. Deep neural networks (DNNs) represent the state-of-the-art models to address real-world classification. Although the strength of activation in DNNs is often correlated with the network's confidence, in-depth analyses are needed to establish whether they are well calibrated.

Method: In this paper, we demonstrate the use of DNN-based classification tools to benefit cancer registries by automating information extraction of disease at diagnosis and at surgery from electronic text pathology reports from the US National Cancer Institute (NCI) Surveillance, Epidemiology, and End Results (SEER) population-based cancer registries. In particular, we introduce multiple methods for selective classification to achieve a target level of accuracy on multiple classification tasks while minimizing the rejection amount-that is, the number of electronic pathology reports for which the model's predictions are unreliable. We evaluate the proposed methods by comparing our approach with the current in-house deep learning-based abstaining classifier.

Results: Overall, all the proposed selective classification methods effectively allow for achieving the targeted level of accuracy or higher in a trade-off analysis aimed to minimize the rejection rate. On in-distribution validation and holdout test data, with all the proposed methods, we achieve on all tasks the required target level of accuracy with a lower rejection rate than the deep abstaining classifier (DAC). Interpreting the results for the out-of-distribution test data is more complex; nevertheless, in this case as well, the rejection rate from the best among the proposed methods achieving 97% accuracy or higher is lower than the rejection rate based on the DAC.

Conclusions: We show that although both approaches can flag those samples that should be manually reviewed and labeled by human annotators, the newly proposed methods retain a larger fraction and do so without retraining-thus offering a reduced computational cost compared with the in-house deep learning-based abstaining classifier.

Keywords: Abstaining classifier; Accuracy; CNN; DNN; Deep learning; HiSAN; NCI SEER; Pathology reports; Selective classification; Text classification; Uncertainty quantification.

Fig. 1.

Subfigure (a) shows the architecture of the MTCNN for multi-task classification. The model has three parallel filters with a different window size for each filter. The output from these filters is fed into a maxpooling layer and is then concatenated before a final softmax or sigmoid function is applied for each classification task. In subfigure (b), the MTHiSAN architecture is presented, showing how the different layers of word embeddings create a word hierarchy which is connected to a self-attention and target attention, respectively. The output of these attention mechanisms is directly connected to similar hierarchical attention mechanisms, creating a hierarchy over the lines in a pathology report. These features create the document embedding, which are the extracted features used in the final classification layer.

Fig. 2.

Baseline accuracy for our models on validation and test data for both in-distribution, more recent holdout data (UTNJKYLASA) as well as OOD data (CA and NM). In all instances, the accuracy is lower than the required 97% level.

Fig. 3.

Experimental study 1: validation data. The accuracy level of about 97% is achieved with the displayed rejection rate. (*) and (x) represent the lowest and highest rejection rate by task.

Fig. 4.

Experimental study 1: more recent, hold-out test set. The tuning on the validation set resulted in higher accuracy on the test set than the target level of 97%, corresponding to the displayed rejection rate. (*) and (x) represent the lowest and highest rejection rate by task.

Fig. 5.

Experimental study 2: validation set (top) and more recent, hold-out test set (bottom). The tuning on the validation set for the same self-tuning accuracy selected by the DAC resulted in a higher accuracy than the target level of 97%. (*) and (x) represent the lowest and highest rejection rate by task. Also, all the proposed a posteriori methods retain a larger or equal rate of classes (i.e., number of retained predicted classes vs. ground truth CTC classes) compared to the DAC.

Fig. 6.

Experimental study 2: OOD test set – CA (top) and NM (bottom). The tuning on the validation set led to a higher accuracy on the test set than the target level of 97%. (*) and (x) represent the lowest and highest rejection rate by task. Also, all the proposed a posteriori methods retain a larger or equal rate of classes (i.e., number of retained predicted classes vs. ground truth CTC classes) compared to the DAC.