CLTNet: A Hybrid Deep Learning Model for Motor Imagery Classification

Abstract

Background: Brain-computer interface (BCI) technology opens up new avenues for human-machine interaction and rehabilitation by connecting the brain to machines. Electroencephalography (EEG)-based motor imagery (MI) classification is a key component of BCI technology, which is capable of translating neural activity in the brain into commands for controlling external devices. Despite the great potential of BCI technology, the challenges of extracting and decoding brain signals limit its wide application.

Methods: To address this challenge, this study proposes a novel hybrid deep learning model, CLTNet, which focuses on solving the feature extraction problem to improve the classification of MI-EEG signals. In the preliminary feature extraction stage, CLTNet uses a convolutional neural network (CNN) to extract time series, channel, and spatial features of EEG signals to obtain important local information. In the deep feature extraction stage, the model combines the long short-term memory (LSTM) network and the Transformer module to capture time-series data and global dependencies in the EEG. The LSTM explains the dynamics of the brain activity, while the Transformer's self-attention mechanism reveals the global features of the time series. Ultimately, the CLTNet model classifies motor imagery EEG signals through a fully connected layer.

Results: The model achieved an average accuracy of 83.02% and a Kappa value of 0.77 on the BCI IV 2a dataset, and 87.11% and a Kappa value of 0.74 on the BCI IV 2b dataset, both of which outperformed the traditional methods.

Conclusions: The innovation of the CLTNet model is that it integrates multiple network architectures, which offers a more comprehensive understanding of the characteristics of the EEG signals during motor imagery, providing a more comprehensive perspective and establishing a new benchmark for future research in this area.

Keywords: brain–computer interface; convolutional neural network; deep learning; long short-term memory network; motor imagery; multi-head attention.

Figure 1

Deep learning architecture for hybrid network models of the CLTNet.

Figure 2

Processes within the motor imagery paradigm (example: BCI IV 2a).

Figure 3

Data enhancement principle.

Figure 4

LSTM module structure.

Figure 5

Transformer encoder architecture.

Figure 6

Average confusion matrices of the proposed CLTNet: (a) the BCI IV-2a dataset and (b) the BCI IV-2b dataset.

Figure 7

ROC curves for different models and their corresponding AUC values: (a) the BCI IV-2a dataset and (b) the BCI IV-2b dataset. (Conformer refers to the EEG Conformer model).