Multi-channel attention-fusion neural network for brain age estimation: Accuracy, generality, and interpretation with 16,705 healthy MRIs across lifespan

Abstract

Brain age estimated by machine learning from T1-weighted magnetic resonance images (T1w MRIs) can reveal how brain disorders alter brain aging and can help in the early detection of such disorders. A fundamental step is to build an accurate age estimator from healthy brain MRIs. We focus on this step, and propose a framework to improve the accuracy, generality, and interpretation of age estimation in healthy brain MRIs. For accuracy, we used one of the largest sample sizes (N = 16,705). For each subject, our proposed algorithm first explicitly splits the T1w image, which has been commonly treated as a single-channel 3D image in other studies, into two 3D image channels representing contrast and morphometry information. We further proposed a "fusion-with-attention" deep learning convolutional neural network (FiA-Net) to learn how to best fuse the contrast and morphometry image channels. FiA-Net recognizes varying contributions across image channels at different brain anatomy and different feature layers. In contrast, multi-channel fusion does not exist for brain age estimation, and is mostly attention-free in other medical image analysis tasks (e.g., image synthesis, or segmentation), where treating channels equally may not be optimal. For generality, we used lifespan data 0-97 years of age for real-world utility; and we thoroughly tested FiA-Net for multi-site and multi-scanner generality by two phases of cross-validations in discovery and replication data, compared to most other studies with only one phase of cross-validation. For interpretation, we directly measured each artificial neuron's correlation with the chronological age, compared to other studies looking at the saliency of features where salient features may or may not predict age. Overall, FiA-Net achieved a mean absolute error (MAE) of 3.00 years and Pearson correlation r=0.9840 with known chronological ages in healthy brain MRIs 0-97 years of age, comparing favorably with state-of-the-art algorithms and studies for accuracy and generality across sites and datasets. We also provided interpretations on how different artificial neurons and real neuroanatomy contribute to the age estimation.

Keywords: Age prediction; Attention network; Deep learning; Lifespan brain MRI; Multi-channel fusion.

Figure 1:

Overview of the proposed network architecture. It has three branches: two channel-specific networks (FiA-Net_con and FiA-Net_mor, top and bottom paths in blue colors) provide channel-specific age estimation results, and one attention-driven fusion network (FiA-Net_fus, middle path in orange) provides the final age estimation results. In channel-specific branches, the Res_i(i = 1, 2, 3, 4) boxes are residual network blocks, and $f_i^{m_i}$ are the intermediate deep features of channel image m_i after the i^th residual block. In the fusion branch, the F_i boxes are the fusion blocks. GAP is the global average pooling.

Figure 2:

Explicit split of a 3D T1w image into two 3D images representing two channels of information (contrast and morphometry). (a) A subject’s T1w image was registered to the SRI24 atlas, leading to a 3D registered intensity image (the first channel, contrast information) and a 3D RAVENS image (the second channel, morphometry information), both residing in the SRI24 atlas space. (b) Randomly-chosen subjects in every ten years of the age range for their two channels of images. Each column shows the MRI slices in the axial (top row), sagittal (middle row), and coronal (bottom row) planes. All images resided in the SRI24 atlas space.

Figure 3:

Three different fusion strategies. Orange arrows represent the fusion.

Figure 4:

The proposed attention-driven fusion block in the i^th layer. (a) Overview of the fusion block, which contains four attention mechanisms: hard attention, illustrated in (b) and the output features are denoted as $f_i^{u_1}$; soft attention, illustrated in (c) and the output features are denoted as $f_i^{u_2}$; and two mutual attentions, illustrated in (d) and the output features are denoted as $f_i^{u_3}$ and $f_i^{u_4}$.

Figure 5:

Our two-phase validation strategy. The first cross-validation happened in the discovery cohort, which was split in five folds of equal sample sizes (Test i) for five cross-validations. In each cross validation, the set ”Test i” (dark orange box) was used for evaluation and the rest four folds (light orange box) were used for training ”Model i” (blue boxes). In the second phase of validation, each trained ”Model i” (blue box) was applied on the completely-unseen replication cohort (green box) to evaluate accuracy and generality.

Figure 6:

Strategy to interpret the predictive value of each neuron in the deep neural network.

Figure 7:

Further understanding of the important choices in our algorithm. The predictive value of the four attention mechanisms and their concatenations ($f_4^{u_1}$, $f_4^{u_2}$, $f_4^{u_3}$, $f_4^{u_4}$, and $f_4^{u_c}$ ) at the last layer of the fusion branch FiA-Net_fus.

Figure 8:

Accuracy comparisons among three algorithms using the same data and the same 5-fold cross-validation strategies, using MAE as the accuracy metric. The solid red line in each panel describes the ideal predictions where the predicted ages are identical to the chronological ages. Each green dot represent a subject.

Figure 9:

Accuracy comparisons among three algorithms using the same data and the same 5-fold cross-validation strategies, using CS as the accuracy metric. The CS curve of brain age estimation using the different networks in cross validation.

Figure 10:

Accuracy comparison among different studies that used different datasets. Each column is one study. They used different datasets and had different age ranges. Red dots, following the red scale bar on the left, are the Mean Absolution Error (MAE) in each study. Blue bars, following the blue scale bar on the right, are the age ranges in each study. Our proposed study is represented by the blue bar on the most right part of the figure. The gray rectangle box highlights four studies, in the right part of the figure, which used > 6,000 subjects and lifespan data. Therefore, these studies in the gray rectangle box are more comparable, and among them, our study had the lowest MAE.

Figure 11:

Interpretation of four most-weighted neurons in the last layer of FiA-Net_fus. (a)-(d): voxel-wise correlations with chronological ages in the four neurons on different age groups. (e): voxel-wise correlations with chronological ages in each neuron over 0–97 years. The neurons were ranked by their weights in the last layer of FiA-Net_fus in descending order. (f): average correlations in 62 auto-segmented brain structures.

Figure 12:

Age estimation errors (MAE) as a function of sample size at each age. The color curves are the MAEs of different algorithms, and they comply with the scales in the left y axis. The gray bars are the numbers of samples at each age, and they follow the scales in the right y axis.