Recognizing Brain States Using Deep Sparse Recurrent Neural Network

Abstract

Brain activity is a dynamic combination of different sensory responses and thus brain activity/state is continuously changing over time. However, the brain's dynamical functional states recognition at fast time-scales in task fMRI data have been rarely explored. In this paper, we propose a novel 5-layer deep sparse recurrent neural network (DSRNN) model to accurately recognize the brain states across the whole scan session. Specifically, the DSRNN model includes an input layer, one fully-connected layer, two recurrent layers, and a softmax output layer. The proposed framework has been tested on seven task fMRI data sets of Human Connectome Project. Extensive experiment results demonstrate that the proposed DSRNN model can accurately identify the brain's state in different task fMRI data sets and significantly outperforms other auto-correlation methods or non-temporal approaches in the dynamic brain state recognition accuracy. In general, the proposed DSRNN offers a new methodology for basic neuroscience and clinical research.

Fig. 1.

A schematic illustration of RNN cell models. (a) The nonlinear basic recurrent cell unit; (b) Long short-term memory unit; (c) Gated recurrent unit; (d) The interconnections in a common recurrent hidden layer. Neurons in recurrent layer are fully interconnected and new hidden states can be influenced by all former states. Squares indicate linear combination and nonlinearity. Circles indicate element wise operations. Gates in the units control the information flow between adjacent time points.

Fig. 2.

Overview of the DSRNN model. The stretched fMRI vector sequences are fed to the input layer of DSRNN model. One fully connected layer is used to extract activated brain regions. Two recurrent layers with dropout are cascaded to model temporal dynamics. Finally, a softmax classifier is arranged for the brain state recognition.

Fig. 3.

Illustration of DSRNN modeling process. (a) Raw brain tfMRI images series. (b) Vectorization of tfMRI image series after sampling. (c) Hierarchy diagram of DSRNN model. (d) Weight matrix between input layer and fully connected layer. (e) Visualization of activated brain region groups represented by weight matrix of fully connected layer. (f) Output time series of fully connected layer. Each column vector indicates the activation of 32 distinctive brain region groups. (g) Visualization of fully time-scale output time series of fully connected layer, also the activation traces of 32 brain region groups. Motor task is used as an example, and the ground-truth temporal distribution of event blocks is illustrated at the top. (h) Visualization of clustered activation traces.(i) Visualization of brain activation maps corresponding to 5 motions, based on 32 brain region groups (e) and their clustered activation traces (h). Brain activation maps from I to VI are corresponding to the motion events of right foot, left foot, tongue, left hand and right hand, respectively. (j) Recognition of brain states.

Fig. 4.

Brain states recognition with recurrent layers, Softmax, SVM and AAR based on activation traces.

Fig. 5.

Brain state recognition accuracies of seven tasks.

Fig. 6.

Brain state series of seven tasks. In each subgraph, five state series from top to bottom are ground truth (GT), and series recognized by DSRNN,AAR, Softmax and SVM, respectively.

Fig. 7.

Activation maps of motor task.

Fig. 8.

The exploration results of hyper parameters. (a) Results of GRUbased network. (b) Results of LSTM-based network. (c) Results of basic unitbased network. The horizontal axis represents the number of neurons in fully connected layer, and the vertical axis indicates the recognition accuracy. All networks contain two recurrent layers, and bars in different colors indicate different numbers of cell units per recurrent layer.

Fig. 9.

The exploration results of DSRNN with different numbers of recurrent layers. The horizontal axis represents the total number of recurrent cell units, and the vertical axis indicates the recognition accuracy. Bars in different colors indicate different recurrent layer numbers.

Fig. 10.

The exploration results of sparseness penalty weights: beta and lambda. The horizontal axis indicates the values of lambda, and the vertical axis denotes the recognition accuracy. Bars in different colors indicate different beta values.

Fig. 11.

The exploration of dropout proportion based on seven tasks. The horizontal axis indicates the proportion, and the vertical axis denotes the recognition accuracy. Curves in different colors indicate different tasks.