Assessing accuracy and consistency in intracranial aneurysm sizing: human expertise vs. artificial intelligence

Abstract

Intracranial aneurysms (IAs) are a common vascular pathology and are associated with a risk of rupture, which is often fatal. Aneurysm growth of more than 1 mm is considered a surrogate of rupture risk, therefore, this study presents a comprehensive analysis of intracranial aneurysm measurements utilizing a dataset comprising 358 IA from 248 computed tomography angiography (CTA) scans measured by four junior raters and one senior rater. The study explores the variability in sizing assessments by employing both human raters and an Artificial Intelligence (AI) system. Our findings reveal substantial inter- and intra-rater variability among junior raters, contrasting with the lower intra-rater variability observed in the senior rater. Standard deviations of all raters were above the threshold for IA growth (1 mm). Additionally, the study identifies a systemic bias, indicating a tendency for human experts to measure aneurysms smaller than the AI system. Our findings emphasize the challenges in human assessment while also showcasing the capacity of AI technology to improve the precision and reliability of intracranial aneurysm assessments, especially beneficial for junior raters. The potential of AI was particularly evident in the task of monitoring IA at various intervals, where the AI-based approach surpassed junior raters and achieved performance comparable to senior raters.

Figure 1

Summary of the study’s four experiments: inter-rater analysis, intra-rater analysis, validation of a computerized approach, and size monitoring.

Figure 2

Inter-rater variability analysis of IA size measurement for each of four junior raters versus the senior rater, with corresponding scatter and Bland-Altman plots (top and bottom rows, respectively).

Figure 3

Intra-rater variability analysis of IA size measurement for each of four junior raters and the senior rater, with corresponding scatter and Bland-Altman plots (top and bottom rows, respectively).

Figure 4

Analysis of manual IA size measurement versus automated measurements for all raters, with corresponding scatter and Bland-Altman plots (top and bottom rows, respectively).

Figure 5

Distribution of the senior rater’s and computerized measurements. Corresponding measurements are indicated by the connecting lines.

Figure 6

Aneurysm size measurements over time for the junior raters (gray lines), senior rater (red lines), and the computerized approach (blue lines).