Predicting cancer outcomes with radiomics and artificial intelligence in radiology

Abstract

The successful use of artificial intelligence (AI) for diagnostic purposes has prompted the application of AI-based cancer imaging analysis to address other, more complex, clinical needs. In this Perspective, we discuss the next generation of challenges in clinical decision-making that AI tools can solve using radiology images, such as prognostication of outcome across multiple cancers, prediction of response to various treatment modalities, discrimination of benign treatment confounders from true progression, identification of unusual response patterns and prediction of the mutational and molecular profile of tumours. We describe the evolution of and opportunities for AI in oncology imaging, focusing on hand-crafted radiomic approaches and deep learning-derived representations, with examples of their application for decision support. We also address the challenges faced on the path to clinical adoption, including data curation and annotation, interpretability, and regulatory and reimbursement issues. We hope to demystify AI in radiology for clinicians by helping them to understand its limitations and challenges, as well as the opportunities it provides as a decision-support tool in cancer management.

Fig. 1 |. Workflow for AI-enabled biomarkers in radiology.

Typical protocol for developing artificial intelligence (AI) radiology biomarkers using radiomic and deep learning approaches, and their clinical applications. Both approaches can be applied in the context of cancer outcome prediction and biomarker discovery for assessment of response to treatment, prognostication and radiogenomics. DICOM, Digital Imaging and Communications in Medicine; ML, machine learning; OS, overall survival; PFS, progression-free survival; RFS, recurrence-free survival.

Fig. 2 |. Examples of the types of radiomic feature used in oncology.

a | Grey-level co-occurrence matrix entropy in a metastatic liver lesion detected by CT. b | Shape of a glioblastoma detected on gadolinium-enhanced T1-weighted MRI. c | Kinetic measure of contrast enhancement over time in breast tissue using contrast-enhanced MRI. d | Peritumoural radiomics measuring textural heterogeneity in the lung stroma surrounding a non-small-cell carcinoma. e | Shape of breast vasculature and tumour-associated vessel network detected using contrast-enhanced MRI. f | Enhanced standardized uptake values on 2-deoxy-2-¹⁸F-fiuoro-D-giucose PET-CT scans showing increased metabolic activity in a head and neck carcinoma.

Fig. 3 |. Building blocks and types of neural network commonly applied to medical imaging data.

a | Example of a convolutional neural network (CNN) model configured for prediction. Input images or volumes are passed through the CNN layers, which perform operations and translate them into a target output vector. Convolutional layers are sets of operations that transform imaging data into deep-feature representations. Each filter is passed over the image and paired with a nonlinear activation function to emphasize visual patterns of interest for a certain task. As more convolutional layers are stacked, a CNN can learn more complex visual patterns within an image. Throughout a CNN classifier, deep features are periodically aggregated through pooling operations. After processing by convolutional and pooling layers, deep-feature representations are eventually flattened into a vector. Next, fully connected layers translate these CNN-derived image features into a vector that corresponds to a target output. These models can be applied to the prediction of treatment response, prognostication, classification of tumour subtypes and biomarkers, and prediction of physiological values. b | Fully convolutional neural networks are a type of CNN comprising only convolutional layers that yield image-like outputs, such as a map of a tumour’s location. c | Fully connected networks can be trained to make predictions based on non-image data, such as radiomic features and clinical variables.

Fig. 4 |. Different levels of annotation detail in radiomics and deep learning studies.

Defining the region of interest and level of annotation detail in radiomics and deep learning studies. TME, tumour microenvironment.