Validation of MRI-Based Models to Predict MGMT Promoter Methylation in Gliomas: BraTS 2021 Radiogenomics Challenge

Abstract

O6-methylguanine-DNA methyl transferase (MGMT) methylation prediction models were developed using only small datasets without proper external validation and achieved good diagnostic performance, which seems to indicate a promising future for radiogenomics. However, the diagnostic performance was not reproducible for numerous research teams when using a larger dataset in the RSNA-MICCAI Brain Tumor Radiogenomic Classification 2021 challenge. To our knowledge, there has been no study regarding the external validation of MGMT prediction models using large-scale multicenter datasets. We tested recent CNN architectures via extensive experiments to investigate whether MGMT methylation in gliomas can be predicted using MR images. Specifically, prediction models were developed and validated with different training datasets: (1) the merged (SNUH + BraTS) (n = 985); (2) SNUH (n = 400); and (3) BraTS datasets (n = 585). A total of 420 training and validation experiments were performed on combinations of datasets, convolutional neural network (CNN) architectures, MRI sequences, and random seed numbers. The first-place solution of the RSNA-MICCAI radiogenomic challenge was also validated using the external test set (SNUH). For model evaluation, the area under the receiver operating characteristic curve (AUROC), accuracy, precision, and recall were obtained. With unexpected negative results, 80.2% (337/420) and 60.0% (252/420) of the 420 developed models showed no significant difference with a chance level of 50% in terms of test accuracy and test AUROC, respectively. The test AUROC and accuracy of the first-place solution of the BraTS 2021 challenge were 56.2% and 54.8%, respectively, as validated on the SNUH dataset. In conclusion, MGMT methylation status of gliomas may not be predictable with preoperative MR images even using deep learning.

Keywords: MRI; O6-methylguanine-DNA methyl transferase; gliomas; neural network; radiogenomics.

Figure 1

Patient inclusion and exclusion criteria. Abbreviations: T1w, T1-weighted imaging; T2w, T2-weighted imaging; T1wCE, contrast-enhanced T1-weighted imaging; FLAIR, fluid-attenuated inversion recovery.

Figure 2

Model performance in the (a) validation (or tuning) and (b) test sets. For each of the experiments, both accuracy and AUROC are shown to report the model performance in the (a) validation (i.e., tuning) and (b) test sets. Note that the dashed red lines are the chance level (50%). The horizontal axis is the dataset on which the model was trained/validated (i.e., trained/tuned): “Public” indicates that the model was trained/validated on the BraTS dataset and tested on the SNUH dataset. “SNUH” indicates that the model was trained/validated on the SNUH dataset and tested on the BraTS dataset. “Merged” indicates that the model was trained/validated and tested with a randomly split SNUH + BraTS dataset. Error bars indicate the standard deviation of the metrics. Note that the validation metrics are better than the test metrics because the model training was stopped early according to the high validation accuracy. The red dotted lines indicate the chance level. Abbreviations: AUROC, area under the receiver operating characteristic curve.

Figure 3

Probability score distribution according to the MGMT labels. The probability scores were predicted from the best model of each experiment (specified in Table 4) and obtained using the test set specific to each experiment. Note that there is no noticeable boundary in the distribution of data points between the groups with high and low probability scores, according to MGMT labels, which are indicated as different colors. MGMT+ (blue) indicates the methylated MGMT promotor group, and MGMT- (orange) indicates the unmethylated MGMT promotor group. The red dotted lines indicate chance level. Abbreviations: MGMT, O6-methylguanine-DNA methyltransferase.