Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis

Abstract

For the last decade, it has been shown that neuroimaging can be a potential tool for the diagnosis of Alzheimer's Disease (AD) and its prodromal stage, Mild Cognitive Impairment (MCI), and also fusion of different modalities can further provide the complementary information to enhance diagnostic accuracy. Here, we focus on the problems of both feature representation and fusion of multimodal information from Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). To our best knowledge, the previous methods in the literature mostly used hand-crafted features such as cortical thickness, gray matter densities from MRI, or voxel intensities from PET, and then combined these multimodal features by simply concatenating into a long vector or transforming into a higher-dimensional kernel space. In this paper, we propose a novel method for a high-level latent and shared feature representation from neuroimaging modalities via deep learning. Specifically, we use Deep Boltzmann Machine (DBM)(2), a deep network with a restricted Boltzmann machine as a building block, to find a latent hierarchical feature representation from a 3D patch, and then devise a systematic method for a joint feature representation from the paired patches of MRI and PET with a multimodal DBM. To validate the effectiveness of the proposed method, we performed experiments on ADNI dataset and compared with the state-of-the-art methods. In three binary classification problems of AD vs. healthy Normal Control (NC), MCI vs. NC, and MCI converter vs. MCI non-converter, we obtained the maximal accuracies of 95.35%, 85.67%, and 74.58%, respectively, outperforming the competing methods. By visual inspection of the trained model, we observed that the proposed method could hierarchically discover the complex latent patterns inherent in both MRI and PET.

Keywords: Alzheimer's Disease; Deep Boltzmann Machine; Mild Cognitive Impairment; Multimodal data fusion; Shared feature representation.

Figure 1

Schematic illustration of the proposed method in hierarchical feature representation and multimodal fusion with deep learning for AD/MCI Diagnosis. (I: image size, w: patch size, K: # of the selected patches, m: modality index, F_G: # of hidden units in a Gaussian restricted Boltzmann machine, i.e., preprocessor, F_S: # of hidden units in the top layer of a multimodal deep Boltzmann machine).

Figure 2

An architecture of a restricted Boltzmann machine (a) and its simplified representation (b).

Figure 3

An architecture of (a) a conventional deep Boltzmann machine and (b) its discriminative version with label information at the top layer.

Figure 4

An architecture of a multimodal deep Boltzmann machine for neuroimaging data fusion.

Figure 5

Visualization of the p-value distributions used to select the patch locations of MRI and PET in AD and NC classification.

Figure 6

Samples of the selected patches, whose voxel values are the input to the (MM-)DBM.

Figure 7

Visualization of the trained weights of our modality-specific Gaussian RBMs (left) used for data conversion from a real-valued vector to a binary vector, and those of our MM-DBM (right) used for latent feature representations. For the weights of our MM-DBM, they correspond to the first hidden layer in the respective modality’s pathway in the model. In each subfigure, one row corresponds to one hidden unit in the respective Gaussian RBM or MM-DBM.

Figure 8

Patch weight distributions in classification of AD vs. NC and MCI vs. NC.