Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy

Abstract

Background and purpose: In radiotherapy, automatic organ-at-risk segmentation algorithms allow faster delineation times, but clinically relevant contour evaluation remains challenging. Commonly used measures to assess automatic contours, such as volumetric Dice Similarity Coefficient (DSC) or Hausdorff distance, have shown to be good measures for geometric similarity, but do not always correlate with clinical applicability of the contours, or time needed to adjust them. This study aimed to evaluate the correlation of new and commonly used evaluation measures with time-saving during contouring.

Materials and methods: Twenty lung cancer patients were used to compare user-adjustments after atlas-based and deep-learning contouring with manual contouring. The absolute time needed (s) of adjusting the auto-contour compared to manual contouring was recorded, from this relative time-saving (%) was calculated. New evaluation measures (surface DSC and added path length, APL) and conventional evaluation measures (volumetric DSC and Hausdorff distance) were correlated with time-recordings and time-savings, quantified with the Pearson correlation coefficient, R.

Results: The highest correlation (R = 0.87) was found between APL and absolute adaption time. Lower correlations were found for APL with relative time-saving (R = -0.38), for surface DSC with absolute adaption time (R = -0.69) and relative time-saving (R = 0.57). Volumetric DSC and Hausdorff distance also showed lower correlation coefficients for absolute adaptation time (R = -0.32 and 0.64, respectively) and relative time-saving (R = 0.44 and -0.64, respectively).

Conclusion: Surface DSC and APL are better indicators for contour adaptation time and time-saving when using auto-segmentation and provide more clinically relevant and better quantitative measures for automatically-generated contour quality, compared to commonly-used geometry-based measures.

Keywords: Added path length; Automatic delineation; Contouring time; Dice similarity coefficient; Hausdorff distance; Radiotherapy; Surface DSC; Time-saving.

Fig. 1

Illustration of evaluation measures used in this study. The solid line represents the automatically generated contour, the dashed line the user-adjusted contour. A Volumetric DSC, defined as the union of two volumes (green volume region) normalized by the mean of the two volumes. B Hausdorff distance, defined as the maximum nearest neighbor Euclidean distance (green arrow). C Surface DSC, defined as the union of two contours (yellow contour region) normalized by the mean surface of the two contours. D Added path length (APL) (yellow contour region). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 2

Example of transverse CT images of two patients showing the automatically generated (solid line) and user-adjusted (dashed line) contours of the heart. For both patients, the volumetric DSC of the automatically generated and user-adjusted contour is similar (94%), but the MSHD, surface DSC, and APL differ between these patients (0.92 cm vs. 1.34 cm, 44% vs. 68%, 678 cm vs. 409 cm, respectively). The contouring time needed to make both contours clinically acceptable for these patients also differed (3.6 min vs. 3.1 min and 6% vs. 20%, respectively). Manual contouring time for both contours was approximately 4 min.

Fig. 3

All measures against absolute time needed to adjust the automatically generated contour, comparing the automatic to the adjusted contour. Atlas-based (circles) and deep-learning contouring (triangles) were combined. MSHD = Mean Slice-wise Hausdorff Distance.

Fig. 4

All measures against relative contouring time saved after adjusting the automatically generated contour, comparing the automatic to the adjusted contour. Atlas-based (circles) and deep-learning contouring (triangles) were combined. MSHD = Mean Slice-wise Hausdorff Distance. One outlier (x = −321%) is not shown in these graphs.

Fig. 5

All measures against absolute time needed to adjust the automatically generated contour, comparing the automatic to the manual contour. Atlas-based (circles) and deep-learning contouring (triangles) were combined. MSHD = Mean Slice-wise Hausdorff Distance.