Deep learning based automatic detection algorithm for acute intracranial haemorrhage: a pivotal randomized clinical trial

Abstract

Acute intracranial haemorrhage (AIH) is a potentially life-threatening emergency that requires prompt and accurate assessment and management. This study aims to develop and validate an artificial intelligence (AI) algorithm for diagnosing AIH using brain-computed tomography (CT) images. A retrospective, multi-reader, pivotal, crossover, randomised study was performed to validate the performance of an AI algorithm was trained using 104,666 slices from 3010 patients. Brain CT images (12,663 slices from 296 patients) were evaluated by nine reviewers belonging to one of the three subgroups (non-radiologist physicians, n = 3; board-certified radiologists, n = 3; and neuroradiologists, n = 3) with and without the aid of our AI algorithm. Sensitivity, specificity, and accuracy were compared between AI-unassisted and AI-assisted interpretations using the chi-square test. Brain CT interpretation with AI assistance results in significantly higher diagnostic accuracy than that without AI assistance (0.9703 vs. 0.9471, p < 0.0001, patient-wise). Among the three subgroups of reviewers, non-radiologist physicians demonstrate the greatest improvement in diagnostic accuracy for brain CT interpretation with AI assistance compared to that without AI assistance. For board-certified radiologists, the diagnostic accuracy for brain CT interpretation is significantly higher with AI assistance than without AI assistance. For neuroradiologists, although brain CT interpretation with AI assistance results in a trend for higher diagnostic accuracy compared to that without AI assistance, the difference does not reach statistical significance. For the detection of AIH, brain CT interpretation with AI assistance results in better diagnostic performance than that without AI assistance, with the most significant improvement observed for non-radiologist physicians.

Fig. 1. Diagnostic performance of reviewers in terms of basic ROC curves for patient-wise AI standalone performance.

In the reader assessment study, the AI-assisted group demonstrated significantly higher diagnostic accuracy in AIH detection compared to the AI-unassisted group in the patient-wise analysis (0.9703 [95% CI: 0.9632, 0.9765] vs. 0.9471 [95% CI: 0.9379, 0.9553], p < 0.0001). Based on subgroup analysis, non-radiologist physicians achieved the greatest benefit in terms of improvement in diagnostic accuracy with AI assistance relative to that for the AI-unassisted group (0.9505 [95% CI: 0.9340, 0.9638] vs. 0.9189 [95% CI: 0.8990, 0.9360], with an improvement of 3.15 [95% CI: 0.86, 5.45], p = 0.0072) for non-radiologist physicians to the level of radiologists without AI assistance (0.9459 [95% CI: 0.9290, 0.9599]). In addition, board-certified radiologists demonstrated a significant improvement in diagnostic accuracy with AI assistance relative to that for the AI-unassisted group (0.9741 [95% CI: 0.9614, 0.9835] vs. 0.9459 [95% CI: 0.9290, 0.9599], with an improvement of 2.82 [95% CI: 1.00, 4.63], p = 0.0025), with an improvement for board-certified radiologists to the level of neuroradiologists without AI assistance (0.9764 [95% CI: 0.9641, 0.9853]). Note. ROC: receiver operating characteristic.

Fig. 2. Diagnostic performance of reviewers in terms of basic ROC curves for slice-wise AI standalone performance.

In the reader assessment study, the AI-assisted group demonstrated a significantly higher diagnostic accuracy in detecting AIH than that of the AI-unassisted group in the slice-wise analysis (0.9581 [95% CI: 0.9569, 0.9592] vs. 0.9522 [95% CI: 0.9509, 0.9534], p < 0.0001). Based on the subgroup analysis, non-radiologist physicians and board-certified radiologists demonstrated a significant improvement in diagnostic accuracy with AI assistance relative to that for the AI-unassisted group (for non-radiologist physicians: 0.9393 [95% CI: 0.9369, 0.9417] vs. 0.9306 [95% CI: 0.9280, 0.9332], with a difference of 0.87 [95% CI: 0.52, 1.22], p < 0.0001, for board-certified radiologists 0.9632 [95% CI: 0.9623, 0.9661] vs. 0.9567 [95% CI: 0.9546, 0.9587], with a difference of 0.75 [95% CI: 0.48, 1.03], p < 0.0001). Note. ROC: receiver operating characteristic.

Fig. 3. Representative images of AIH detection.

a AI-assisted brain CT revealed probable AIH location as the basal cistern and right ambient cistern. AI-assisted brain CT provided AIH probability scores in a slice-wise (95.8%) and patient-wise (99.4%) manner. All nine reviewers agreed with the AIH diagnosis for both AI-unassisted and AI-assisted interpretations. b AI-assisted brain CT revealed the probable AIH location as the left side of the falx. AI-assisted brain CT provided the AIH probability scores in a slice-wise (62.2%) and patient-wise (95.3%) manner. For interpretation without AI assistance, one reviewer (non-radiologist physician) missed this case of AIH on the left side of the falx. All nine reviewers agreed with the AIH diagnosis for both AI-unassisted and AI-assisted interpretations. c AI-assisted brain CT revealed probable AIH location as the left parietal sulci. AI-assisted brain CT provided AIH probability scores in a slice-wise (39.0%) and patient-wise (46.3%) manner. For interpretation without AI assistance, two-thirds of the reviewers (three non-radiologist physicians, two board-certified radiologists, and one neuroradiologist) missed this case of AIH in the left parietal sulci. With the use of AI assistance, these six reviewers were able to correctly revise their decisions. d AI-assisted brain CT revealed the probable AIH location as the left frontal area. AI-assisted brain CT provided the AIH probability scores in a slice-wise (54.9%) and patient-wise (65.8%) manner. For the interpretation without AI assistance, one-third of the reviewers (one non-radiologist physician and two board-certified radiologists) reported it as an AIH. With the use of AI assistance, an additional one-third of the reviewers (one non-radiologist physicians, one board-certified radiologist, and one neuroradiologist) reported this as an AIH. However, the subtle hyperattentuating lesion in the left frontal area was due to the beam-hardening artefact of the skull.

Fig. 4. Overview of the AI algorithm.

The diagram shows the architecture of the proposed AI algorithm. This new AI algorithm combined a supervised haemorrhage detection process and an unsupervised anomaly detection process. In addition, a combined CNN-RNN architecture was applied in the haemorrhage detection process. The presence or absence is determined through the haemorrhage detection process. As a result of this haemorrhage detection process, the AI algorithm provides the AIH score in the patient-wise and slice-wise manner. The AI algorithm provides the anomaly map for AIH patients through the subtraction between the original CT image and restored CT image (artificially generated normal image based on the unsupervised training from normal dataset) and postprocessing. The average additional time to access the AI-assisted CT images on PACS viewer was 97.4 seconds. Time from PACS server to AI, AI processing time, and time from AI to PACS viewer were 54.6 seconds (range, 37–91 seconds), 11.8 seconds (range, 0.8–90.6 seconds), 31.0 seconds (range, 30–33 seconds). Note. AIH acute intracranial haemorrhage, PACS picture archiving and communication system, CNN convolutional neural network, RNN recurrent neural network, VAE variational autoencoder, GAN generative adversarial network.

Fig. 5. Schematic overview of study design.

The schematic diagram shows the retrospective, pivotal, crossover, randomised study design used in the present study (left). In the first image review, group A consisted of original CT images and corresponding AI-assisted CT images, while group B consisted of only the original CT images without AI-assisted CT images. After a washout period of 4–5 weeks, in the second image review, the group A dataset was changed to include only the original CT images without any AI-assisted CT images, while AI-assisted CT images were added to the group B dataset. The AI-assisted CT images provided a heatmap with information on the suspected location and probability of AIH in a patient- and slice-wise manner (right).