Time-frequency-space transformer EEG decoding for spinal cord injury

Abstract

Transformer neural networks based on multi-head self-attention are effective in several fields. To capture brain activity on electroencephalographic (EEG) signals and construct an effective pattern recognition model, this paper explores the multi-channel deep feature decoding method utilizing the self-attention mechanism. By integrating inter-channel features with intra-channel features, the self-attention mechanism generates a deep feature vector that encompasses information from all brain activities. In this paper, a time-frequency-spatial domain analysis of motor imagery (MI) based EEG signals from spinal cord injury patients is performed to construct a transformer neural network-based MI classification model. The proposed algorithm is named time-frequency-spatial transformer. The time-frequency and spatial domain feature vectors extracted from the EEG signals are input into the transformer neural network for multiple self-attention depth feature encoding, a peak classification accuracy of 93.56% is attained through the fully connected layer. By constructing the attention matrix brain network, it can be inferred that the channel connections constructed by the attention heads have similarities to the brain networks constructed by the EEG raw signals. The experimental results reveal that the self-attention coefficient brain network holds significant potential for brain activity analysis. The self-attention coefficient brain network can better illustrate correlated connections and show sample differences. Attention coefficient brain networks can provide a more discriminative approach for analyzing brain activity in clinical settings.

Keywords: Brain networks; Motor imagery; Self-attention; Spinal cord injury; Transformer.

Fig. 1

The proposed TFS–TR model structure. The TFS–TR relies on the Transformer network to decode the MST and CSP features to get the intention of the subject’s motor phenomena and to analyze the patient’s brain activity through the self-attention coefficient brain network

Fig. 2

64 electrodes distribution

Fig. 3

Experimental paradigm. Subjects were out of resting state and received MI cues for 0–4 s and MI for 4–7 s

Fig. 4

Transformer and TransformerBlock structure. a is the structure of the Transformer neural network and b is the structure of the TransformerBlock

Fig. 5

The loss curve of Transformer model

Fig. 6

Spatial distribution of model features. The yellow and purple dots in the figure represent EEG features for different categories of MI. As the model goes deeper, the feature vectors are reconstructed with differentiability

Fig. 7

Illustration of the highlight clustering at different locations on the attention coefficient map

Fig. 8

TransformerBlock attention coefficient map. The posture of the five different attention coefficient maps is marked by different colored boxes

Fig. 9

Spatial distribution of TransformerBlock1 and TransformerBlock2 channel feature vectors. The figure shows the distribution of electrode feature vectors in two-dimensional space for three regions: forebrain electrodes (mainly including frontal electrodes), midbrain electrodes (mainly including parietal and temporal electrodes) and hindbrain electrodes (mainly including occipital electrodes). Blue dots are forebrain electrodes, orange dots are midbrain electrodes, and green are hindbrain electrodes

Fig. 10

PLV brain network and attention factor coefficients network.The dots in the figure represent the EEG electrodes, and the lines represent the connection between the two electrodes with synchronized brain activity