Effect of time windows in LSTM networks for EEG-based BCIs

Abstract

People with impaired motor function could be helped by an effective brain-computer interface (BCI) based on a real-time electroencephalogram (EEG) and artificial intelligence algorithms. However, current methodologies for interpreting patient instructions from an EEG are not accurate enough to be completely safe in a real-world situation , where a poor decision would place their physical integrity at risk, such as when traveling in an electric wheelchair in a city. For various reasons, such as the low signal-to-noise ratio of portable EEGs or the effects of signal contamination (disturbances due to user movement, temporal variation of the features of EEG signals, etc.), a long short-term memory network (LSTM) (a type of recurrent neural network) that is able to learn data flow patterns from EEG signals could improve the classification of the actions taken by the user. In this paper, the effectiveness of using an LSTM with a low-cost wireless EEG device in real time is tested, and the time window that maximizes its classification accuracy is studied. The goal is to be able to implement it in the BCI of a smart wheelchair with a simple coded command protocol, such as opening or closing the eyes, which could be executed by patients with reduced mobility. Results show a higher resolution of the LSTM with an accuracy range between 77.61 and 92.14% compared to traditional classifiers (59.71%), and an optimal time window of around 7 s for the task done by users in this work. In addition, tests in real-life contexts show that a trade-off between accuracy and response times is necessary to ensure detection.

Keywords: Artificial neural network; Brain–computer interface; Deep learning; EEG; LSTM; Machine learning.

Fig. 1

Flowchart for EEG signal pre-processing, feature extraction and classification

Fig. 2

Location of the Emotiv EPOC + headset electrodes as per the 10–20 system, where CMS and DRL are references electrodes at P3 and P4, respectively

Fig. 3

Experimental protocol. Each trial consisted of a 20-s fixation period with eyes open and a cross on the screen, 10-s rest with a black screen and 20-s period with eyes closed. A beep marks when to close or open the eyes. EEG data were recorded during the entire time

Fig. 4

Denoising example for eye-blink. The black lines represent the EEG signals recorded at positions AF3, AF4, F7 and F8 when an eye-blink occurs. The gray lines are the same recordings without the eye-blink artefact after applying the REBLINCA method (Di Flumeri et al. 2016) adapted to the Emotiv EPOC + device

Fig. 5

Threshold example application on a segment of closed-eyes state. The left topographic map corresponds to the power before the application of the threshold, and the right one to the power after it

Fig. 6

Dataset generation for LSTM network, showing an example of time-varying alpha-band power records recorded at position O1 (black trace) and O2 (gray trace) used by the LSTM algorithm. The power traces are segmented (sequences 1, 2,…, x) for use by the LSTM, by taking the power acquired in a time window, starting backwards from the point in question

Fig. 7

Architecture of the neural network used in this work. The LSTM layer has cells, the structure of which is shown in the exploded view, where x_t is the input, h_t the output and c_t is the state

Fig. 8

Example of the temporal variation of the power of the alpha band in positions O1 and O2 when the eyes are open or closed

Fig. 9

Classification results with different algorithms obtained for one user. A Power in the alpha band recorded with eyes open and closed in O1 and O2 position. Eyes are closed for 20 s. B Classification results for the DTL, SVM, KNN and RNN algorithms with the configurations that yielded the best training accuracy (Table 1) and an LSTM classifier with a time window of 7 s

Fig. 10

Classification results for LSTM with time windows of 2, 4 and 7 s for eyes closed time periods of 7, 15 and 20 s

Fig. 11

Representation of the different accuracies (y-axis) obtained with various neural networks having different numbers of LSTM layers (each layer defines 8 neurons). Windows between 1 and 10 s have been applied (x-axis)