Explainable fMRI-based brain decoding via spatial temporal-pyramid graph convolutional network

Abstract

Brain decoding, aiming to identify the brain states using neural activity, is important for cognitive neuroscience and neural engineering. However, existing machine learning methods for fMRI-based brain decoding either suffer from low classification performance or poor explainability. Here, we address this issue by proposing a biologically inspired architecture, Spatial Temporal-pyramid Graph Convolutional Network (STpGCN), to capture the spatial-temporal graph representation of functional brain activities. By designing multi-scale spatial-temporal pathways and bottom-up pathways that mimic the information process and temporal integration in the brain, STpGCN is capable of explicitly utilizing the multi-scale temporal dependency of brain activities via graph, thereby achieving high brain decoding performance. Additionally, we propose a sensitivity analysis method called BrainNetX to better explain the decoding results by automatically annotating task-related brain regions from the brain-network standpoint. We conduct extensive experiments on fMRI data under 23 cognitive tasks from Human Connectome Project (HCP) S1200. The results show that STpGCN significantly improves brain-decoding performance compared to competing baseline models; BrainNetX successfully annotates task-relevant brain regions. Post hoc analysis based on these regions further validates that the hierarchical structure in STpGCN significantly contributes to the explainability, robustness and generalization of the model. Our methods not only provide insights into information representation in the brain under multiple cognitive tasks but also indicate a bright future for fMRI-based brain decoding.

Keywords: brain decoding; brain-inspired models; cognitive tasks; fMRI; graph neural networks; human connectome project; model explainability.

FIGURE 1

An overview of our framework. (a) Graph construction: The brain is parcellated into N ROIs by brain atlas, and the functional connectivity (FC) between ROIs are calculated by Pearson correlation on resting‐state fMRI for each individual. The brain graph $\mathcal{G}$ is constructed with top‐k brain graph construction strategy which performs on the binarized averaged functional connectivity W. (b) Brain decoding: STpGCN takes the preprocessed parcellated individual BOLD signals time series from fMRI as input to classify 23 task‐related brain states. (c) Brain annotation: BrainNetX together with the well‐trained STpGCN annotates the core brain regions involved in each task‐related brain state.

FIGURE 2

Confusion matrix of 23 task‐related brain states classification on 15 time points of fMRI with MMP atlas using STpGCN.

FIGURE 3

Classification accuracy (%) of different lengths of fMRI with MMP atlas. The time points (lengths) vary from 4 to 15 (repetition time Tr = 0.72). The results of GCN, GAT, ST‐GCN, and STpGCN are shown in blue, yellow, orange, and red, respectively. The number and error bar indicate the mean and std of test accuracy across 10‐fold cross‐validation.

FIGURE 4

Explainability of STpGCN. We visualize the importance score of 23 task‐related brain states using STpGCN given 15 time points of fMRI with MMP atlas. The importance score is averaged across 10‐fold cross‐validation.

FIGURE 5

Explainability of MLP‐Mixer. We visualize the importance score of 23 task‐related brain states using MLP‐Mixer given 15 time points of fMRI with MMP atlas. The importance score is averaged across 10‐fold cross‐validation.

FIGURE 6

Visualization of the averaged importance score of different working memory tasks with the same stimuli by STpGCN and BrainNetX given 15 time points of fMRI with MMP atlas across 10‐fold cross‐validation.