Reproducibility analysis of multi-institutional paired expert annotations and radiomic features of the Ivy Glioblastoma Atlas Project (Ivy GAP) dataset

Abstract

Purpose: The availability of radiographic magnetic resonance imaging (MRI) scans for the Ivy Glioblastoma Atlas Project (Ivy GAP) has opened up opportunities for development of radiomic markers for prognostic/predictive applications in glioblastoma (GBM). In this work, we address two critical challenges with regard to developing robust radiomic approaches: (a) the lack of availability of reliable segmentation labels for glioblastoma tumor sub-compartments (i.e., enhancing tumor, non-enhancing tumor core, peritumoral edematous/infiltrated tissue) and (b) identifying "reproducible" radiomic features that are robust to segmentation variability across readers/sites.

Acquisition and validation methods: From TCIA's Ivy GAP cohort, we obtained a paired set (n = 31) of expert annotations approved by two board-certified neuroradiologists at the Hospital of the University of Pennsylvania (UPenn) and at Case Western Reserve University (CWRU). For these studies, we performed a reproducibility study that assessed the variability in (a) segmentation labels and (b) radiomic features, between these paired annotations. The radiomic variability was assessed on a comprehensive panel of 11 700 radiomic features including intensity, volumetric, morphologic, histogram-based, and textural parameters, extracted for each of the paired sets of annotations. Our results demonstrated (a) a high level of inter-rater agreement (median value of DICE ≥0.8 for all sub-compartments), and (b) ≈24% of the extracted radiomic features being highly correlated (based on Spearman's rank correlation coefficient) to annotation variations. These robust features largely belonged to morphology (describing shape characteristics), intensity (capturing intensity profile statistics), and COLLAGE (capturing heterogeneity in gradient orientations) feature families.

Data format and usage notes: We make publicly available on TCIA's Analysis Results Directory (https://doi.org/10.7937/9j41-7d44), the complete set of (a) multi-institutional expert annotations for the tumor sub-compartments, (b) 11 700 radiomic features, and (c) the associated reproducibility meta-analysis.

Potential applications: The annotations and the associated meta-data for Ivy GAP are released with the purpose of enabling researchers toward developing image-based biomarkers for prognostic/predictive applications in GBM.

Keywords: IvyGAP; MRI; glioblastoma; radiomics; reproducibility; segmentation.

FIG. 1.

Overall workflow of the present work.

FIG. 2.

Screenshots of Subject W8, showcasing the various registration transformations between CWRU and UPenn annotation we have used in Section 2.D, with the corresponding overall DICE scores. Green represents the UPenn tumor annotations and red represents the CWRU transformed annotations.

FIG. 3.

Inter-rater agreement using three-dimensional volumetric analysis comparing CWRU rater with UPenn rater using different metrics (DICE, Sensitivity, Specificity, Hausdorff) across labels. Note that Specificity has also been plotted on a magnified scale to better highlight differences between the various sub-compartments.

FIG. 4.

Subject W26, showing screenshots of the axial, sagittal and coronal views, where the points of entry for prior instrumentation are visible and highlighted with a red square in the image.

FIG. 5.

Screenshots of Subject W50, where the raters’ agreement regarding the site of NET and ET was different (locations with largest diameter of Non-enhancing part of tumor highlighted for each annotation).

FIG. 6.

Inter-rater agreement analysis using Spearman’s rank correlation coefficient for the UPenn and CWRU raters across the 8 feature families, as well as across T1, T1Gd, T2, and FLAIR protocols.

FIG. 7.

Thermometer plot highlighting the percentage of robust features across UPenn and CWRU segmentations, (with Spearman’s correlation coefficient of ≥0.8) for the 8 feature families across T1, T1Gd, T2, FLAIR protocols.

FIG. 8.

Screenshots of Subject W8, showcasing maximal agreement between UPenn and CWRU raters (with regard to the whole tumor). Each image shows the axial slice from all 4 structural modalities in the top row with the annotations of UPenn and CWRU raters in the bottom 2 rows.