"Real-world" radiomics from multi-vendor MRI: an original retrospective study on the prediction of nodal status and disease survival in breast cancer, as an exemplar to promote discussion of the wider issues

Abstract

Background: Most MRI radiomics studies to date, even multi-centre ones, have used "pure" datasets deliberately accrued from single-vendor, single-field-strength scanners. This does not reflect aspirations for the ultimate generalisability of AI models. We therefore investigated the development of a radiomics signature from heterogeneous data originating on six different imaging platforms, for a breast cancer exemplar, in order to provide input into future discussions of the viability of radiomics in "real-world" scenarios where image data are not controlled by specific trial protocols but reflective of routine clinical practice.

Methods: One hundred fifty-six patients with pathologically proven breast cancer underwent multi-contrast MRI prior to neoadjuvant chemotherapy and/or surgery. From these, 92 patients were identified for whom T2-weighted, diffusion-weighted and contrast-enhanced T1-weighted sequences were available, as well as key clinicopathological variables. Regions-of-interest were drawn on the above image types and, from these, semantic and calculated radiomics features were derived. Classification models using a variety of methods, both with and without recursive feature elimination, were developed to predict pathological nodal status. Separately, we applied the same methods to analyse the information carried by the radiomic features regarding the originating scanner type and field strength. Repeated, ten-fold cross-validation was employed to verify the results. In parallel work, survival modelling was performed using random survival forests.

Results: Prediction of nodal status yielded mean cross-validated AUC values of 0.735 ± 0.15 (SD) for clinical variables alone, 0.673 ± 0.16 (SD) for radiomic features only, and 0.764 ± 0.16 (SD) for radiomics and clinical features together. Prediction of scanner platform from the radiomics features yielded extremely high values of AUC between 0.91 and 1 for the different classes examined indicating the presence of confounding features for the nodal status classification task. Survival analysis, gave out-of-bag prediction errors of 19.3% (clinical features only), 36.9-51.8% (radiomic features from different combinations of image contrasts), and 26.7-35.6% (clinical plus radiomics features).

Conclusions: Radiomic classification models whose predictive ability was consistent with previous single-vendor, single-field strength studies have been obtained from multi-vendor, multi-field-strength data, despite clear confounding information being present. However, our sample size was too small to obtain useful survival modelling results.

Keywords: Feature reduction; Multi-vendor; Nodal status; Radiomics; Survival.

Fig. 1

(a) Flow diagram of subject exclusion process; (b) Venn diagram illustrating availability of data between image contrast types and explaining the patient numbers in the right-hand side of (a)

Fig. 2

Pseudo-code describing model fitting, parameter tuning and performance estimating using a nested cross-validation process, as described in the text

Fig. 3

An exemplar image set showing both original ROIs and the repeat annotations for the ICC feature stability sub-study for (a) T2w-weighted (b) early-phase dynamic subtraction, and (c) diffusion-weighted images

Fig. 4

Radiomics features selected on the basis of the intraclass correlation coefficient (ICC), using a two-way “agreement” model, with threshold of 0.75, for the three different imaging contrasts

Fig. 5

Mean ROC curves for nodal status classification problem using a Naïve Bayes classifier

Fig. 6

Analysis of the composition of models produced using recursive feature elimination: variable importance averaged across model folds and repetitions for models involving predictors drawn from (a) clinical data, (b) radiomics data (calculated plus semantic features, (c) clinical and radiomic data

Fig. 7

Principal component plot for the imaging feature data for all patients, with data points colour-coded by MR scanner type. Larger symbols represent group centroids. This partial separation via unsupervised classification methods demonstrates the significant extent to which data source acts as a confounding factor in radiomics studies of real-world data

Fig. 8

Mean variable importance for the top 10 variables in a Naïve Bayes fitted model to classify images by the manufacturer of scanner on which they were acquired, as an illustration of the degree to which scanner type is a confounding factor influencing the radiomics features

Fig. 9

Kaplan-Meier plots for the survival data showing censoring events and separation of strata by: (a) nodal disease status (NDS); (b) tumour grade; (c) all combinations of nodal status and grade. Quoted p-values are for the null hypothesis that the survival curves for the given strata are the same. It will be seen that almost all death events come from the group that has nodal involvement and tumour grade 3