Effects of a comprehensive brain computed tomography deep learning model on radiologist detection accuracy

Abstract

Objectives: Non-contrast computed tomography of the brain (NCCTB) is commonly used to detect intracranial pathology but is subject to interpretation errors. Machine learning can augment clinical decision-making and improve NCCTB scan interpretation. This retrospective detection accuracy study assessed the performance of radiologists assisted by a deep learning model and compared the standalone performance of the model with that of unassisted radiologists.

Methods: A deep learning model was trained on 212,484 NCCTB scans drawn from a private radiology group in Australia. Scans from inpatient, outpatient, and emergency settings were included. Scan inclusion criteria were age ≥ 18 years and series slice thickness ≤ 1.5 mm. Thirty-two radiologists reviewed 2848 scans with and without the assistance of the deep learning system and rated their confidence in the presence of each finding using a 7-point scale. Differences in AUC and Matthews correlation coefficient (MCC) were calculated using a ground-truth gold standard.

Results: The model demonstrated an average area under the receiver operating characteristic curve (AUC) of 0.93 across 144 NCCTB findings and significantly improved radiologist interpretation performance. Assisted and unassisted radiologists demonstrated an average AUC of 0.79 and 0.73 across 22 grouped parent findings and 0.72 and 0.68 across 189 child findings, respectively. When assisted by the model, radiologist AUC was significantly improved for 91 findings (158 findings were non-inferior), and reading time was significantly reduced.

Conclusions: The assistance of a comprehensive deep learning model significantly improved radiologist detection accuracy across a wide range of clinical findings and demonstrated the potential to improve NCCTB interpretation.

Clinical relevance statement: This study evaluated a comprehensive CT brain deep learning model, which performed strongly, improved the performance of radiologists, and reduced interpretation time. The model may reduce errors, improve efficiency, facilitate triage, and better enable the delivery of timely patient care.

Key points: • This study demonstrated that the use of a comprehensive deep learning system assisted radiologists in the detection of a wide range of abnormalities on non-contrast brain computed tomography scans. • The deep learning model demonstrated an average area under the receiver operating characteristic curve of 0.93 across 144 findings and significantly improved radiologist interpretation performance. • The assistance of the comprehensive deep learning model significantly reduced the time required for radiologists to interpret computed tomography scans of the brain.

Keywords: Artificial intelligence; Brain; Machine learning; Supervised machine learning; Tomography, x-ray computed.

Fig. 1

Change in AUC of parent findings when radiologists were assisted by the deep learning model. Mean AUCs of the model, unassisted, and assisted radiologists and change in (i.e. delta) AUC, along with adjusted 95% CIs, are shown for each parent finding. Findings were considered clinically significant where the lower limit of the 95% CI was greater than 0.05, and statistically significant where the lower limit of the 95% CI was greater than zero

Fig. 2

ROC curves for the parent findings demonstrating the performance of the model, and the mean performance of the assisted and unassisted radiologists

Fig. 3

Performance improvement using the deep learning model. Precision and recall (i.e. sensitivity) for the unassisted and assisted radiologists averaged across all findings, based on the chosen beta levels for each finding. Arrows indicate the shift in recall and precision of the radiologists when assisted by AI. On average, model assistance resulted in increased recall (sensitivity) with no decrement in precision

Fig. 4

A Non-contrast CT brain study of a 79-year-old female who presented with acute stroke symptoms. Subtle hypodensity in the right occipital lobe was missed by 30 of the 32 readers in the unassisted arm of the study, but detected by 26 readers when using the deep learning tool as an assistant. B Output of the model. The model accurately localized the large area of infarction within the right occipital lobe (purple shading). C DWI image clearly showing the area of acute infarction in this patient. D An example of small bilateral isodense subacute subdural haematomas. E The haematomas were characterized by the model as subacute subdural haematomas and localized with purple shading. F A CT scan performed 7 days later. The haematoma is more conspicuous on the later scan as it evolves to become hypodense. G Non-contrast CT brain study demonstrating a colloid cyst. H The same colloid cyst case along with an example of the model’s segmentation and high confidence

Fig. 5

Non-contrast CT brain study demonstrating an intraventricular haemorrhage, along with an example of the decision support system’s user interface

Fig. 6

A three-dimensional (3D) visualisation of a single case containing multiple clinical findings demonstrating the 3D functionality of the model. The findings predicted by the model are presented alongside the ground-truth