The Brain Tumor Segmentation - Metastases (BraTS-METS) Challenge 2023: Brain Metastasis Segmentation on Pre-treatment MRI

Abstract

The translation of AI-generated brain metastases (BM) segmentation into clinical practice relies heavily on diverse, high-quality annotated medical imaging datasets. The BraTS-METS 2023 challenge has gained momentum for testing and benchmarking algorithms using rigorously annotated internationally compiled real-world datasets. This study presents the results of the segmentation challenge and characterizes the challenging cases that impacted the performance of the winning algorithms. Untreated brain metastases on standard anatomic MRI sequences (T1, T2, FLAIR, T1PG) from eight contributed international datasets were annotated in stepwise method: published UNET algorithms, student, neuroradiologist, final approver neuroradiologist. Segmentations were ranked based on lesion-wise Dice and Hausdorff distance (HD95) scores. False positives (FP) and false negatives (FN) were rigorously penalized, receiving a score of 0 for Dice and a fixed penalty of 374 for HD95. The mean scores for the teams were calculated. Eight datasets comprising 1303 studies were annotated, with 402 studies (3076 lesions) released on Synapse as publicly available datasets to challenge competitors. Additionally, 31 studies (139 lesions) were held out for validation, and 59 studies (218 lesions) were used for testing. Segmentation accuracy was measured as rank across subjects, with the winning team achieving a LesionWise mean score of 7.9. The Dice score for the winning team was 0.65 ± 0.25. Common errors among the leading teams included false negatives for small lesions and misregistration of masks in space. The Dice scores and lesion detection rates of all algorithms diminished with decreasing tumor size, particularly for tumors smaller than 100 mm3. In conclusion, algorithms for BM segmentation require further refinement to balance high sensitivity in lesion detection with the minimization of false positives and negatives. The BraTS-METS 2023 challenge successfully curated well-annotated, diverse datasets and identified common errors, facilitating the translation of BM segmentation across varied clinical environments and providing personalized volumetric reports to patients undergoing BM treatment.

Keywords: Artificial Intelligence; BraTS; BraTS-METS; Brain metastasis; Brain tumor segmentation; Machine learning; Medical image analysis challenge.

Figure 1:

Flow chart outlining the BraTS-METS 2023 vision, beginning with the pre-treatment BMs segmentation during the 2023 ASNR/MICCAI BraTS challenge. In this phase, segmentations were conducted on a select dataset subset to refine the dataset for algorithm development by participants. The dataset is set to expand in subsequent challenges through ongoing annotation of contributed brain MRIs. Future challenges will incorporate datasets with annotated post-treatment BMs, segmentations including the hemorrhagic component of tumors, and non-skull-stripped images to enhance the evaluation of dural-based and osseous metastases. These datasets, coupled with clinical data and patient demographics, will contribute to an inter-institutional BMs consortium, fostering collaborative research and the clinical application of algorithms through partnerships between academia and industry.

Figure 2:

Image panels illustrating the annotated tumor sub-regions across various mpMRI scans with segmentations of ET (yellow), SNFH (green), and NETC (red) done on ITK-SNAP.

Figure 3:

BraTS-METS 2023 annotation pipeline.

Figure 4:

Map of institutions that expressed interest in contributing data to the BraTS-METS challenge.

Figure 5:

BraTS-METS 2023 boxplots of LesionWise ranking across patients for all participating teams on the BraTS 2023 test set (lower is better).

Figure 6:

BraTS-METS 2023 boxplots of enhancing tumor Dice scores (A) and 95% Hausdorff distance (HD95) (B) for all participating teams on the BraTS 2023 test set.

Figure 7:

BraTS-METS 2023 boxplots of tumor core Dice scores (A) and 95% Hausdorff distance (HD95) (B) for all participating teams on the BraTS 2023 test set.

Figure 8:

BraTS-METS 2023 boxplots of whole tumor Dice scores (A) and 95% Hausdorff distance (HD95) (B) for all participating teams on the BraTS 2023 test set.

Figure 9:

Performance metrics across tumor entities—whole tumor (WT), tumor core (TC), and enhancing tumor (ET).

Figure 10:

BraTS-METS 2023 plot of cumulative average of (A) Dice scores, (B) 95% Hausdorff distance (HD95), and (C) lesion detection rate as a function of increasing lesion volume.