Multi-modal data fusion using source separation: Two effective models based on ICA and IVA and their properties

Abstract

Fusion of information from multiple sets of data in order to extract a set of features that are most useful and relevant for the given task is inherent to many problems we deal with today. Since, usually, very little is known about the actual interaction among the datasets, it is highly desirable to minimize the underlying assumptions. This has been the main reason for the growing importance of data-driven methods, and in particular of independent component analysis (ICA) as it provides useful decompositions with a simple generative model and using only the assumption of statistical independence. A recent extension of ICA, independent vector analysis (IVA) generalizes ICA to multiple datasets by exploiting the statistical dependence across the datasets, and hence, as we discuss in this paper, provides an attractive solution to fusion of data from multiple datasets along with ICA. In this paper, we focus on two multivariate solutions for multi-modal data fusion that let multiple modalities fully interact for the estimation of underlying features that jointly report on all modalities. One solution is the Joint ICA model that has found wide application in medical imaging, and the second one is the the Transposed IVA model introduced here as a generalization of an approach based on multi-set canonical correlation analysis. In the discussion, we emphasize the role of diversity in the decompositions achieved by these two models, present their properties and implementation details to enable the user make informed decisions on the selection of a model along with its associated parameters. Discussions are supported by simulation results to help highlight the main issues in the implementation of these methods.

Fig. 1

IVA for multi-dataset analysis and the two key signal properties available in addition to HOS: sample dependence and dependence among sources within a source component matrix S_n

Fig. 2

Two examples to demonstrate the role of diversity for (a) ICA (HOS and sample dependence); and (b) IVA (HOS, sample dependence, and source dependence) with respect to the induced CRLB. For both examples, note the improvement in performance as the role of HOS increases, i.e., as shape parameter β moves away from 1, as sample dependence, i.e., the value of AR coefficient a, increases, and in the case of IVA, as shown in (b), as source dependence measured by correlation across datasets ρ increases.

Fig. 3

Two models for multi-modal data fusion for multiple datasets—shown for K = 2. Note the change in the role of sources and mixing matrix columns for the profiles and components.

Fig. 4

Generative model for the simulations.

Fig. 5

Estimation performance for the common component with two datasets as (a) the height of the step in the profile changes, (b) the noise level added to the step profile changes, and (c) the number of subjects increases with a fixed noise and height level.

Fig. 6

Estimation performance for the common component with three datasets as the connection of one dataset—the height of the step—decreases in terms of (a) t-statistic for the most significant component, (b) correlation of the most significant component to the original, and (c) when the connection of one dataset is kept low and the number of subjects increases.