Need for objective task-based evaluation of AI-based segmentation methods for quantitative PET

Abstract

Artificial intelligence (AI)-based methods are showing substantial promise in segmenting oncologic positron emission tomography (PET) images. For clinical translation of these methods, assessing their performance on clinically relevant tasks is important. However, these methods are typically evaluated using metrics that may not correlate with the task performance. One such widely used metric is the Dice score, a figure of merit that measures the spatial overlap between the estimated segmentation and a reference standard (e.g., manual segmentation). In this work, we investigated whether evaluating AI-based segmentation methods using Dice scores yields a similar interpretation as evaluation on the clinical tasks of quantifying metabolic tumor volume (MTV) and total lesion glycolysis (TLG) of primary tumor from PET images of patients with non-small cell lung cancer. The investigation was conducted via a retrospective analysis with the ECOG-ACRIN 6668/RTOG 0235 multi-center clinical trial data. Specifically, we evaluated different structures of a commonly used AI-based segmentation method using both Dice scores and the accuracy in quantifying MTV/TLG. Our results show that evaluation using Dice scores can lead to findings that are inconsistent with evaluation using the task-based figure of merit. Thus, our study motivates the need for objective task-based evaluation of AI-based segmentation methods for quantitative PET.

Keywords: Task-based evaluation; artificial intelligence; positron emission tomography; quantification; segmentation.

Figure 1.

Architecture of the considered commonly used U-net-based convolutional neural network model.

Figure 2.

Task-based evaluation of the impact of network depth on the performance of the considered CNN model. (A) Performance quantified using the task-agnostic Dice scores and task-based figure of merit of absolute ensemble normalized bias; (B) Violin plot showing the distributions of the Dice scores and normalized error in estimated MTV/TLG for a shallower and deeper network, respectively.

Figure 3.

Segmentations of the primary tumor in a representative patient obtained with a shallower and a deeper network. The segmentations for each of the slices that contained the tumor are shown. For this patient, the Dice scores were very similar between the shallower and deeper network, even though the errors in estimating MTV/TLG were very different. We also note that visually, the segmentations are different for the shallower vs. deeper networks.