How to develop a meaningful radiomic signature for clinical use in oncologic patients

Abstract

During the last decade, there is an increasing usage of quantitative methods in Radiology in an effort to reduce the diagnostic variability associated with a subjective manner of radiological interpretation. Combined approaches where visual assessment made by the radiologist is augmented by quantitative imaging biomarkers are gaining attention. Advances in machine learning resulted in the rise of radiomics that is a new methodology referring to the extraction of quantitative information from medical images. Radiomics are based on the development of computational models, referred to as "Radiomic Signatures", trying to address either unmet clinical needs, mostly in the field of oncologic imaging, or to compare radiomics performance with that of radiologists. However, to explore this new technology, initial publications did not consider best practices in the field of machine learning resulting in publications with questionable clinical value. In this paper, our effort was concentrated on how to avoid methodological mistakes and consider critical issues in the workflow of the development of clinically meaningful radiomic signatures.

Keywords: Machine learning; Quantitative imaging; Radiomics.

Fig. 1

A multidisciplinary radiomics workflow. Initially a group of clinicians should define the clinical problem that the proposed model should deal with and make decisions on what kind of imaging modalities should be recruited. Imaging scientists needs to make sure that acquisition protocols are optimally designed producing high quality images, as well as for the pre-processing of the images. Then depending on the size of the available imaging studies we need to decide which pipeline to use. In case of big data (in the order of thousands) a deep radiomics approach can be suggested avoiding tedious and time-consuming processes like tumor segmentation by multiple radiologists. In addition, deep convolutional neural networks have been proven more efficient to model complex problems compared with traditional machine learning algorithms, as long as data availability requirement is satisfied. Finally, the data sets are allocated for training, validation and testing purposes

Fig. 2

Feature selection is accomplished by applying several methods in a cascade manner. A typical workflow in the first phase permits only stable features to be forwarded, then a zero or near-zero variance method is removing useless features, then a correlation analysis is removing redundant features and finally a more sophisticated method like maximum relevance minimum redundancy (mRMR) or recursive feature elimination (RFE) is used to craft the final Radiomic signature

Fig. 3

A convolutional neural network (VNet architecture) was trained on arterial phase images of a dynamic contrast enhanced MRI dataset to automatically segment enhancing breast lesions. Two examples are shown (the worst and the best) with an average DICE coefficient of 0.82 ± 0.15. The pink color denotes the pixels that where considered from the network as a lesion while the white pixels where corresponding to the radiologists’ segmentation used as the ground truth. The DICE coefficient is defined as 2 * the Area of Overlap between the pink and white areas divided by the total number of pixels in the segmentation mask. 130 patients in total were recruited for the training of the model

Fig. 4

Correlation analysis heatmap showing blocks of highly correlated radiomic features (black frames on the left and positive with red color or negative correlation with blue color on the right). When identifying such groups of highly correlated features all but the one with the highest variance are removed from further analysis. In this case, the correlation coefficient was set to 95%

Fig. 5

A heatmap aggregating the performance results of combinations of 6 machine learning models and 9 feature selection techniques. The dataset used for this analysis comprised features extracted from malignant pancreatic neoplasms on diffusion-weighted MRI acquired at high b value images (? of b = 900 s/mm²), which is used to distinguish patients with synchronous liver metastases from those without metastases. The best performing combination was an LDA model with mRMR feature selection method. SVM: Support Vector Machine, GLM: General Linear Model, LDA: Linear Discriminant Analysis, LG: Logistic Regression, NB: Naïve Bayes, KNN: K Nearest Neighbor, FSCR: Fisher Score, TSCR: T-Score, CHSQ: CHI-Square, WLCX: Wilcoxon, Gini: Gini index, MIM: Mutual Information Maximization, mRMR: minimum Redundancy Maximum Relevance, JMI: Joint mutual information

Fig. 6

Resampling scheme based on the total dataset and iterative multi-splitting scheme based on a 5-fold cross validation. Apart the performance metric (mean AUC in that case) it is of equal importance to report the standard deviation across the folds to get an indication of robustness of the model. Low standard deviations are reflecting stable and robust models that are not influenced by the specific test set