Precise Image-level Localization of Intracranial Hemorrhage on Head CT Scans with Deep Learning Models Trained on Study-level Labels

Abstract

Purpose To develop a highly generalizable weakly supervised model to automatically detect and localize image-level intracranial hemorrhage (ICH) by using study-level labels. Materials and Methods In this retrospective study, the proposed model was pretrained on the image-level Radiological Society of North America dataset and fine-tuned on a local dataset by using attention-based bidirectional long short-term memory networks. This local training dataset included 10 699 noncontrast head CT scans in 7469 patients, with ICH study-level labels extracted from radiology reports. Model performance was compared with that of two senior neuroradiologists on 100 random test scans using the McNemar test, and its generalizability was evaluated on an external independent dataset. Results The model achieved a positive predictive value (PPV) of 85.7% (95% CI: 84.0, 87.4) and an area under the receiver operating characteristic curve of 0.96 (95% CI: 0.96, 0.97) on the held-out local test set (n = 7243, 3721 female) and 89.3% (95% CI: 87.8, 90.7) and 0.96 (95% CI: 0.96, 0.97), respectively, on the external test set (n = 491, 178 female). For 100 randomly selected samples, the model achieved performance on par with two neuroradiologists, but with a significantly faster (P < .05) diagnostic time of 5.04 seconds per scan (vs 86 seconds and 22.2 seconds for the two neuroradiologists, respectively). The model's attention weights and heatmaps visually aligned with neuroradiologists' interpretations. Conclusion The proposed model demonstrated high generalizability and high PPVs, offering a valuable tool for expedited ICH detection and prioritization while reducing false-positive interruptions in radiologists' workflows. Keywords: Computer-Aided Diagnosis (CAD), Brain/Brain Stem, Hemorrhage, Convolutional Neural Network (CNN), Transfer Learning Supplemental material is available for this article. © RSNA, 2024 See also the commentary by Akinci D'Antonoli and Rudie in this issue.

Keywords: Brain/Brain Stem; Computer-Aided Diagnosis (CAD); Convolutional Neural Network (CNN); Hemorrhage; Transfer Learning.

Graphical abstract

Figure 1:

Flowchart for all patients included in this study and the breakdown of pretraining, training, validation, and held-out and external test datasets. ICH = intracranial hemorrhage, NLP = natural language processing, RSNA = Radiological Society of North America.

Figure 2:

The end-to-end deep learning workflow of the model. Stage I pretrained the EfficientNet-B2 network on the RSNA dataset at the image level. Stage II utilized a bidirectional LSTM and an attention module to train the model on our institution’s dataset at the study level. Finally, in stage III, the complete model was fine-tuned into an end-to-end model on our institution’s dataset. Fully-conn represents a fully connected layer and Leakyrelu represents the Leaky Rectified Linear Unit activation function. ICH = intracranial hemorrhage, LSTM = long short-term memory, RSNA = Radiological Society of North America.

Figure 3:

Gradient-weighted class activation map (Grad-CAM)–generated heatmaps to highlight important features for predicting intracranial hemorrhage (ICH) and to visualize the possible regions of ICH. In each figure part, the raw CT image is on the left, and the heatmap generated by Grad-CAM is on the right. (A, C) Examples from our institution’s test set, not annotated by radiologists. The color map (right) indicates possible ICH regions. (B, D) Examples from our institution’s test set, manually annotated by radiologists, where the red mask (left) represents the ICH, and the color map (right) indicates possible ICH regions.

Figure 4:

Illustration of the interpretability of our weakly supervised model for localizing ICH at the image and pixel level. The model was trained at the study level, but the attention weights indicate the probability, from 0 to 1, of each image showing ICH (column 5). The model generated heatmaps on predicted ICH images (designated by red arrows in column 5) to localize specific ICH regions (shown in column 4). The heatmaps were converted to a pixel-level mask and quantified by Dice scores (column 3), which were computed by comparison with the reference standard segments (column 2) manually annotated by radiologists (column 1). Grad-CAM = gradient-weighted class activation map, ICH = intracranial hemorrhage.

Figure 5:

(A–C) Confusion matrix plots for comparisons between the reference standard and predictions or diagnoses from (A) the proposed model, (B) neuroradiologist 1 (Neurorad-1), and (C) neuroradiologist 2 (Neurorad-2). (D) Bar chart demonstrates the percentage of agreement between the model and each neuroradiologist, as well as between the two neuroradiologists. ICH = intracranial hemorrhage.