Radiology and multi-scale data integration for precision oncology

Abstract

In this Perspective paper we explore the potential of integrating radiological imaging with other data types, a critical yet underdeveloped area in comparison to the fusion of other multi-omic data. Radiological images provide a comprehensive, three-dimensional view of cancer, capturing features that would be missed by biopsies or other data modalities. This paper explores the complexities and challenges of incorporating medical imaging into data integration models, in the context of precision oncology. We present the different categories of imaging-omics integration and discuss recent progress, highlighting the opportunities that arise from bringing together spatial data on different scales.

Fig. 1. Types of data integration.

a Simplified illustration of the three types of data integration processes considered in this review, classified as a function of the task and its objective. Data sources of different modalities are depicted in the figure with the letters A-D. Clinically-relevant endpoints are indicated by the label 'Outcome'. b Representative examples of the different levels of data fusion based on the physical scale at which the data is aligned, including patient-level (data sources are treated as independent), lesion-level (imaging features are matched to corresponding samples), or tissue-level (tissue samples are taken from specific locations using image guidance) fusion. c Different types of data fusion architectures in terms of the stage at which data streams are combined. The figure uses radiology and pathology images as an illustration. Clinically-relevant endpoints are indicated by the label 'Outcome'.

Fig. 2. Review of data fusion.

a Breakdown of the papers identified according to whether they satisfy the inclusion criteria; use deep learning or not; and integrate three or more data types or not. b Number of data types integrated in each of the studies that satisfied all the inclusion criteria. c Circos plot illustrating the co-occurrence of the different data types in the studies that satisfied all the inclusion criteria. Prot. = Proteomics, Transcript. = Transcriptomics.

Fig. 3. Schematic of data translation process.

From left to right: types of data samples extracted from tumours, spatial integration techniques for sample co-registration, and data integration techniques for cross-modal translation, including image-to-image translation and image-to-genotype prediction. ST spatial transcriptomics.

Fig. 4. Data aggregation.

Schematic of the data aggregation process to connect clinical information from Electronic Health Records (EHRs), imaging modalities from Picture Archiving and Communication Systems (PACS), and molecular markers from Laboratory Information Management Systems (LIMS), including a standardisation step for interoperability. Data can be processed offline after anonymisation (green pipelines) or displayed and visualised in real time (pink pipelines). Models trained on anonymised data can be deployed and integrated as part of live visualisation tools.