A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer

Abstract

Background and purpose: Clinical Artificial Intelligence (AI) implementations lack ground-truth when applied on real-world data. This study investigated how combined geometrical and dose-volume metrics can be used as performance monitoring tools to detect clinically relevant candidates for model retraining.

Materials and methods: Fifty patients were analyzed for both AI-segmentation and planning. For AI-segmentation, geometrical (Standard Surface Dice 3 mm and Local Surface Dice 3 mm) and dose-volume based parameters were calculated for two organs (bladder and anorectum) to compare AI output against the clinically corrected structure. A Local Surface Dice was introduced to detect geometrical changes in the vicinity of the target volumes, while an Absolute Dose Difference (ADD) evaluation increased focus on dose-volume related changes. AI-planning performance was evaluated using clinical goal analysis in combination with volume and target overlap metrics.

Results: The Local Surface Dice reported equal or lower values compared to the Standard Surface Dice (anorectum: (0.93 ± 0.11) vs (0.98 ± 0.04); bladder: (0.97 ± 0.06) vs (0.98 ± 0.04)). The ADD metric showed a difference of (0.9 ± 0.8)Gy for the anorectum ${\text{D}}_{{1\text{cm}}^3}$. The bladder ${\text{D}}_{{\text{5cm}}^3}$ reported a difference of (0.7 ± 1.5)Gy. Mandatory clinical goals were fulfilled in 90 % of the DLP plans.

Conclusions: Combining dose-volume and geometrical metrics allowed detection of clinically relevant changes, applied to both auto-segmentation and auto-planning output and the Local Surface Dice was more sensitive to local changes compared to the Standard Surface Dice. This monitoring is able to evaluate AI behavior in clinical practice and allows candidate selection for active learning.

Keywords: Artificial intelligence; Clinical metrics; Deep Learning Planning; Deep Learning Segmentation; Performance monitoring; SBRT prostate.

Fig. 1

Schematic overview of a deep learning guided contouring and dose planning workflow. The input is the CT image on which the DLS model generates AI contours R_DL, followed by a radiation oncologist’s review resulting in contours R_clin. Based on R_clin, the DLP model creates plan P_DL which can be further optimized by a planner leading to plan P_clin within the fine-tune optimization step. All datasets (R_DL, R_clin, P_DL, P_clin) are stored in the database and were used for DLS and DLP monitoring.

Fig. 2

Typical output of DLS and DLP models. White, DLS generated ROIs are plotted next to the clinical correct yellow bladder and brown anorectum. On the sagittal plane, the dashed red lines mark the boundaries for the Local Surface Dice calculation. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 3

On the x-axis, Absolute Dose Difference (ADD) for anorectum and bladder were reported for different critical volumes. On the y-axis, the Local Surface Dice is plotted and the color of each dot resembles the Standard Surface Dice as shown by the color bar. The dashed lines resemble the ADD of 1 Gy and the first standard deviation of the Local Surface Dice. Each graph is divided into four zones, reflecting perfect DLS segmentations in zone 1, minor corrected DLS ROIs with large dose-volume impact (zone 2), major corrected DLS ROIs outside the high dose region and less clinical impact in zone 3 and zone 4 shows major corrections with large difference in ADD.

Fig. 4

Clinical goal analysis comparing DLP output against the clinical plan for both bladder (D5cm³ < 37 Gy), anorectum (D1cm³ < 36 Gy) and PTV (V36.25 Gy > 95 %) and every colored dot resembles a patient ROI belonging to one of the four clinical goal evaluation groups. The blue distribution plots above and to the right of each scatter plot show the corresponding data of the training data. The dashed blue lines show the first standard deviation of this training data. The lower left graph visualizes the difference in PTV volume between local and training data. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)