Feature-based fusion of medical imaging data

Abstract

The acquisition of multiple brain imaging types for a given study is a very common practice. There have been a number of approaches proposed for combining or fusing multitask or multimodal information. These can be roughly divided into those that attempt to study convergence of multimodal imaging, for example, how function and structure are related in the same region of the brain, and those that attempt to study the complementary nature of modalities, for example, utilizing temporal EEG information and spatial functional magnetic resonance imaging information. Within each of these categories, one can attempt data integration (the use of one imaging modality to improve the results of another) or true data fusion (in which multiple modalities are utilized to inform one another). We review both approaches and present a recent computational approach that first preprocesses the data to compute features of interest. The features are then analyzed in a multivariate manner using independent component analysis. We describe the approach in detail and provide examples of how it has been used for different fusion tasks. We also propose a method for selecting which combination of modalities provides the greatest value in discriminating groups. Finally, we summarize and describe future research topics.

Fig. 1

Illustration of model in which loading parameters are shared among features. The feature matrix is organized by placing the features (e.g., SPM map and GM map) from the two modalities side by side (with one row containing data collected from the same subject for both modalities). This matrix is then modeled as containing spatially independent joint source images that share common mixing matrix parameters.

Fig. 2

Cross-task 2-D histograms for AOD versus SB fMRI activation: joint 2-D histograms for voxels identified in the analysis. Individual (a) and group average difference (b) histograms [with orange areas larger in controls and blue areas larger in patients] are provided along with the marginal histograms for the AOD (SPM contrast image for “targets”) (c) and Sternberg (SPM contrast image for “recall”) (d) data.

Fig. 3

AOD/GM jICA analysis: only one component demonstrated a significant difference between patients and controls. The joint source map for the AOD (left) and GM (middle) data is presented along with the loading parameters for patients and controls (far right).

Fig. 4

ERP/fMRI jICA: joint component that showed significantly different loading parameters (p < 0.0001) for patients versus controls. Control (yellow) and patient (blue) average ERP plots along with the ERP part of the identified joint component (pink) (left). Thresholded fMRI part of the joint component showing bilateral temporal and frontal lobe regions (right).

Fig. 5

ERP/fMRI histograms: joint histograms for patients (blue) and controls (orange) (left). Simulated data from two Gaussians (a) showing a case in which marginal histograms (b) and (c) are less able to detect differences between groups whereas the histogram in the direction of maximal separation (d) clearly shows the two distributions from patients and controls (right).

Fig. 6

GM and FA spatial correspondence (left) and corresponding weights (right) comparing older and younger participants (p < 0.002).

Fig. 7

Example of evaluation of KL divergence for combinations of features comparing two-way fusion with nonfusion. Larger KL divergence values represent better patient versus control separation.