Inconsistency of AI in intracranial aneurysm detection with varying dose and image reconstruction

Abstract

Scanner-related changes in data quality are common in medical imaging, yet monitoring their impact on diagnostic AI performance remains challenging. In this study, we performed standardized consistency testing of an FDA-cleared and CE-marked AI for triage and notification of intracranial aneurysms across changes in image data quality caused by dose and image reconstruction. Our assessment was based on repeated examinations of a head CT phantom designed for AI evaluation, replicating a patient with three intracranial aneurysms in the anterior, middle and posterior circulation. We show that the AI maintains stable performance within the medium dose range but produces inconsistent results at reduced dose and, unexpectedly, at higher dose when filtered back projection is used. Data quality standards required for AI are stricter than those for neuroradiologists, who report higher aneurysm visibility rates and experience performance degradation only at substantially lower doses, with no decline at higher doses.

Keywords: Artificial intelligence; Computed tomography angiography; Imaging phantoms; Intracranial aneurysm; Reproducibility of findings.

Fig. 1

Setup and CT images of the phantom. Photograph shows the phantom positioned in the CT scanner for image acquisition. CT images of the phantom visualize the three aneurysms in the anterior communicating artery (ACoA), middle cerebral artery (MCA), and basilar artery (BA).

Fig. 2

Examples of AI output and corresponding CT images. First row: The AI-generated heatmap displays true positive labels for aneurysms in the anterior communicating artery (ACoA) and middle cerebral artery (MCA), but not for the basilar artery (BA). Second row: The heatmap indicates a true positive label for the MCA aneurysm, but not for the ACoA and BA aneurysms. Third row: A false negative summary report generated by the AI solution, where BA stands for brain aneurysm. IR = iterative reconstruction, FBP = filtered back projection.

Fig. 3

True positive aneurysm labels in heatmaps generated by the AI solution. Graphs display the absolute numbers of true positive aneurysm labels across three repeated CT examinations per dose and image reconstruction method. IR = iterative reconstruction, FBP = filtered back projection, MCA = middle cerebral artery, ACoA = anterior communicating artery, BA = basilar artery, CTDI_vol = computed tomography dose index.

Fig. 4

Sizes and Intensities of aneurysm labels. Graphs present the size (upper row) and intensity (lower row) of aneurysm labels generated by the AI solution in three repeated CT examinations per dose and image reconstruction method. Dotted lines indicate mean values. IR = iterative reconstruction, FBP = filtered back projection, MCA = middle cerebral artery, ACoA = anterior communicating artery, BA = basilar artery, CTDI_vol = computed tomography dose index.

Fig. 5

Heatmaps generated by the AI solution with positive labels for the basilar artery aneurysm superimposed on corresponding CT images. Labels in CT examinations at 1.01 mGy (A) and 2.02 mGy (B) are slightly offset. (C) The label in a CT examination at 5.02 mGy aligns with the aneurysm. All CT examinations presented were reconstructed using iterative reconstruction.

Fig. 6

Results of aneurysm assessment by five neuroradiologists. Upper row: Absolute number of CT examinations in which aneurysms were rated as visible across three repeated CT examinations per dose and image reconstruction method. Lower row: Image quality ratings per aneurysm, dose, and image reconstruction method. Mean results of the five readers are presented; error bars indicate standard errors of the mean. IR = iterative reconstruction, FBP = filtered back projection, MCA = middle cerebral artery, ACoA = anterior communicating artery, BA = basilar artery, CTDI_vol = computed tomography dose index.