Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Jan 19;23(1):8.
doi: 10.1186/s12938-024-01204-4.

Multimodal diagnosis model of Alzheimer's disease based on improved Transformer

Yan Tang  1   2 , Xing Xiong  3 , Gan Tong  3 , Yuan Yang  4 , Hao Zhang  5
Affiliations

Multimodal diagnosis model of Alzheimer's disease based on improved Transformer

Yan Tang et al. Biomed Eng Online. .

Abstract

Purpose: Recent technological advancements in data acquisition tools allowed neuroscientists to acquire different modality data to diagnosis Alzheimer's disease (AD). However, how to fuse these enormous amount different modality data to improve recognizing rate and find significance brain regions is still challenging.

Methods: The algorithm used multimodal medical images [structural magnetic resonance imaging (sMRI) and positron emission tomography (PET)] as experimental data. Deep feature representations of sMRI and PET images are extracted by 3D convolution neural network (3DCNN). An improved Transformer is then used to progressively learn global correlation information among features. Finally, the information from different modalities is fused for identification. A model-based visualization method is used to explain the decisions of the model and identify brain regions related to AD.

Results: The model attained a noteworthy classification accuracy of 98.1% for Alzheimer's disease (AD) using the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. Upon examining the visualization results, distinct brain regions associated with AD diagnosis were observed across different image modalities. Notably, the left parahippocampal region emerged consistently as a prominent and significant brain area.

Conclusions: A large number of comparative experiments have been carried out for the model, and the experimental results verify the reliability of the model. In addition, the model adopts a visualization analysis method based on the characteristics of the model, which improves the interpretability of the model. Some disease-related brain regions were found in the visualization results, which provides reliable information for AD clinical research.

Keywords: 3DCNN; Alzheimer’s disease; Deep learning; Multimodal medical images; Transformer; Visualization.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no conflict of interest.

Figures

Fig. 1
Fig. 1
Permutation distribution of the estimate
Fig. 2
Fig. 2
ROC curves of different methods
Fig. 3
Fig. 3
Clustering results (cluster size > 100) of remaining pixels (top 1%) in MRI (top) and PET (bottom) images
Fig. 4
Fig. 4
The architecture of deep 3D CNNs denoted with the sizes of each layer’s input, convolution, max pooling, and output layers and the numbers and sizes of generated feature maps. C is a convolutional layer, the P is max pooling layer, @ is the number of filters such as 15@ 3 × 3 × 3 is 15 filters whose size are 3 × 3 × 3 and P 2 × 2 × 2 is pooling layers, with a size of 2 × 2 × 2. The number below each layer represents the shape of the feature
Fig. 5
Fig. 5
The struct of transformer encoder
Fig. 6
Fig. 6
Convolution-based self-attention mechanism
Fig. 7
Fig. 7
The framework of network model
Fig. 8
Fig. 8
Visualization framework
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Gauthier S, Webster C, Servaes S, Morais J, Rosa-Neto P. World Alzheimer report 2022: life after diagnosis: navigating treatment, care and support. London: Alzheimer’s Disease International London; 2022.
    1. Javeed A, Dallora AL, Berglund JS, Anderberg P. An intelligent learning system for unbiased prediction of dementia based on autoencoder and adaboost ensemble learning. Life. 2022;12(7):1097. doi: 10.3390/life12071097. - DOI - PMC - PubMed
    1. Loddo A, Buttau S, Di Ruberto C. Deep learning based pipelines for Alzheimer’s disease diagnosis: a comparative study and a novel deep-ensemble method. Comput Biol Med. 2022;141:105032. doi: 10.1016/j.compbiomed.2021.105032. - DOI - PubMed
    1. Shoeibi A, Khodatars M, Jafari M, Ghassemi N, Moridian P, Alizadesani R, Ling SH, Khosravi A, Alinejad-Rokny H, Lam H. Diagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: a review. Inf Fus. 2022;93:85–117. doi: 10.1016/j.inffus.2022.12.010. - DOI
    1. McEvoy LK, Fennema-Notestine C, Roddey JC, Hagler DJ, Jr, Holland D, Karow DS, Pung CJ, Brewer JB, Dale AM. Alzheimer disease: quantitative structural neuroimaging for detection and prediction of clinical and structural changes in mild cognitive impairment. Radiology. 2009;251(1):195–205. doi: 10.1148/radiol.2511080924. - DOI - PMC - PubMed

MeSH terms