Performance analysis and knowledge-based quality assurance of critical organ auto-segmentation for pediatric craniospinal irradiation

Abstract

Craniospinal irradiation (CSI) is a vital therapeutic approach utilized for young patients suffering from central nervous system disorders such as medulloblastoma. The task of accurately outlining the treatment area is particularly time-consuming due to the presence of several sensitive organs at risk (OAR) that can be affected by radiation. This study aimed to assess two different methods for automating the segmentation process: an atlas technique and a deep learning neural network approach. Additionally, a novel method was devised to prospectively evaluate the accuracy of automated segmentation as a knowledge-based quality assurance (QA) tool. Involving a patient cohort of 100, ranging in ages from 2 to 25 years with a median age of 8, this study employed quantitative metrics centered around overlap and distance calculations to determine the most effective approach for practical clinical application. The contours generated by two distinct methods of atlas and neural network were compared to ground truth contours approved by a radiation oncologist, utilizing 13 distinct metrics. Furthermore, an innovative QA tool was conceptualized, designed for forthcoming cases based on the baseline dataset of 100 patient cases. The calculated metrics indicated that, in the majority of cases (60.58%), the neural network method demonstrated a notably higher alignment with the ground truth. Instances where no difference was observed accounted for 31.25%, while utilization of the atlas method represented 8.17%. The QA tool results showed that the two approaches achieved 100% agreement in 39.4% of instances for the atlas method and in 50.6% of instances for the neural network auto-segmentation. The results indicate that the neural network approach showcases superior performance, and its significantly closer physical alignment to ground truth contours in the majority of cases. The metrics derived from overlap and distance measurements have enabled clinicians to discern the optimal choice for practical clinical application.

Figure 1

Heatmap showing the difference between neural network and atlas overlap metrics to determine if the atlas or neural network is closer to ground truth, designated by blue and red, respectively. L and R subscripts mean left and right, respectively. DSC dice similarity coefficient, JAC Jaccard index, TPR true positive rate, TNR true negative rate, RI rand index, PPV positive predictive value.

Figure 2

Heatmap showing the difference between neural network and atlas distance metrics to determine if the atlas or neural network is closer to ground truth, designated by blue and red, respectively. L and R subscripts mean left and right, respectively. HD Hausdorff distance, MDA mean distance to agreement.

Figure 3

Histograms showing the dice similarity coefficient (DSC) and positive predictive value (PPV) for atlas and neural network methods in 100 patients averaged across all 16 organs.

Figure 4

Average baseline KDE distributions for all 16 organs of 100 CSI patients. L and R subscripts mean left and right, respectively.

Figure 5

Baseline KDE distributions with ± 2 SD (green shaded area) for 16 organs of 100 CSI patients. L and R subscripts mean left and right, respectively.

Figure 6

Percentage agreements of atlas and neural network methods against baseline KDEs for 10 test patients and 16 organs. L and R subscripts mean left and right, respectively.

Figure 7

Single axial CT slice shows the overlaps of yellow line (ground truth), blue line (atlas), and red line (neural network) methods for right kidney in (a). KDEs show percent agreements between the methods against the baseline KDE for right kidney for patient #4 in (b).