Neural correlates of user learning during long-term BCI training for the Cybathlon competition

Abstract

Background: Brain-computer interfaces (BCIs) are systems capable of translating human brain patterns, measured through electroencephalography (EEG), into commands for an external device. Despite the great advances in machine learning solutions to enhance the performance of BCI decoders, the translational impact of this technology remains elusive. The reliability of BCIs is often unsatisfactory for end-users, limiting their application outside a laboratory environment.

Methods: We present the analysis on the data acquired from an end-user during the preparation for two Cybathlon competitions, where our pilot won the gold medal twice in a row. These data are of particular interest given the mutual learning approach adopted during the longitudinal training phase (8 months), the long training break in between the two events (1 year) and the demanding evaluation scenario. A multifaceted perspective on long-term user learning is proposed: we enriched the information gathered through conventional metrics (e.g., accuracy, application performances) by investigating novel neural correlates of learning in different neural domains.

Results: First, we showed that by focusing the training on user learning, the pilot was capable of significantly improving his performance over time even with infrequent decoder re-calibrations. Second, we revealed that the analysis of the within-class modifications of the pilot's neural patterns in the Riemannian domain is more effective in tracking the acquisition and the stabilization of BCI skills, especially after the 1-year break. These results further confirmed the key role of mutual learning in the acquisition of BCI skills, and particularly highlighted the importance of user learning as a key to enhance BCI reliability.

Conclusion: We firmly believe that our work may open new perspectives and fuel discussions in the BCI field to shift the focus of future research: not only to the machine learning of the decoder, but also in investigating novel training procedures to boost the user learning and the stability of the BCI skills in the long-term. To this end, the analyses and the metrics proposed could be used to monitor the user learning during training and provide a marker guiding the decoder re-calibration to maximize the mutual adaptation of the user to the BCI system.

Keywords: Brain-computer interface; Cybathlon; Long-term evaluation; Motor imagery; Mutual learning; Riemann geometry; User learning.

Fig. 1

Overview of the BCI implementation and training protocol. a BCI pipeline to classify both hands and both feet motor imagery. First, the raw EEG signals were spatially filtered and their power spectral density (PSD) extracted. During the offline calibration, the most discriminative features were identified through canonical variate analysis (CVA) and used to calibrate the decoder to classify the two mental tasks. The BCI commands were then converted into the proper game commands to control the BrainDriver game. During the online evaluation only, continuous feedback about the decoder outputs were visually provided to the user to foster learning. b Timeline illustrating the pilot training protocol and the approximate day of decoder update from the first contact with the pilot to the day of the Cybathlon 2020 Global Edition. Between the end of the Cybathlon 2019 BCI Series (17/09/2019) to the following training session (15/09/2020), the pilot spent almost one year without using any BCI system

Fig. 2

Schematic illustration of the metrics proposed to track user learning. The between-class distance represents the distance between the means of the EEG features distribution of the two motor imagery classes (i.e., both hands, both feet). The within-class distance is computed separately for the two classes as the distance of the means of the EEG features distribution with respect to the first day of training. In the channels’ domain the two metrics were calculated using the Euclidean distance, while in the Riemann domain we considered the geodesic distance (i.e., the shortest path between feature distributions following the Riemannian manifold $\mathcal{M}$)

Fig. 3

Cybathlon BCI race completion time. a Race completion times (s) achieved by our pilot throughout training. Training effect is shown by the linear fit and the Pearson correlation coefficient (significance tested with Student t test distribution). Dashed horizontal line illustrate the maximum race completion time allowed during the competition. Vertical thin lines indicate the date of each racing session, while vertical thick black lines represent the dates of decoder update. The break of 1 year is marked by a vertical red line. Markers colored in green and red show the race completion times obtained during the two competitions, in the 2019 BCI Series and the 2020 Global Edition respectively. b Boxplot of race completion times (s) in the first and last 15 races of 2019 and 2020 training periods. The box edges signify the 75th (top) and 25th (bottom) percentiles and the horizontal line the median of the corresponding distribution. The whiskers extend to the largest and smallest nonoutlier values. Single-race values are marked with filled circles. Statistically significant differences are shown with Tukey-Kramer post-hoc test. c Boxplot of section crossing time (s, time spent on each section) in the first 15 races of 2019 and the last 15 races of 2020 training periods. The box edges signify the 75th (top) and 25th (bottom) percentiles and the horizontal line the median of the corresponding distribution. The whiskers extend to the largest and smallest nonoutlier values. Outliers are marked with black crosses, while single-section values with filled circles. Statistically significant differences are shown with two-sided Wilcoxon ranksum tests. * $p<.05$, ** $p<.01$, *** $p<.001$

Fig. 4

BCI performance and topographic maps. a Evolution over training runs of the decoder accuracy (green, % of correctly classified samples) and rejection (red, % of samples whose prediction was discarded due to low confidence). Their corresponding linear fits and Pearson correlation coefficients (significance tested with Student t test distribution) were evaluated for the two years (2019, 2020) separately. Vertical thin lines indicate the date of each training session, while vertical thick black lines represent the dates of decoder update. The break of 1 year is marked by a vertical red line. (b-c) Boxplots of decoder accuracy b and rejection c in the first and last 15 runs of 2019 and 2020 training periods. The box edges signify the 75th (top) and 25th (bottom) percentiles and the horizontal line the median of the corresponding distribution. The whiskers extend to the largest and smallest nonoutlier values. Single-run values are marked with filled circles. Statistically significant differences are shown with Tukey-Kramer post-hoc tests. d Topographic maps of discriminancy per training month on the 14 EEG channel locations over the sensorimotor cortex. Bright color indicates high discriminancy between Both Hands and Both Feet MI tasks. The discriminancy of each channel is quantified as the Fisher score of the EEG signal’s power spectral density distributions for the two mental classes in the $\beta$-band (16–26 Hz) within each run. Each map illustrates local Fisher scores (with interchannel interpolation) averaged over all runs within the month

Fig. 5

Between-class distance in channels’ and Riemann domains. a–c Evolution over races of the between-class distance in channels’ domain a and Riemann domain c computed in the $\mu$-band (8–12 Hz) and $\beta$-band (16–26 Hz). Their corresponding linear fits and Pearson correlation coefficients (significance tested with Student t test distribution) were evaluated for the two years (2019, 2020) separately. Vertical thin lines indicate the date of each racing session, while vertical thick black lines represent the dates of decoder update. The break of 1 year is marked by a vertical red line. b–d Boxplots of between-class distance in channels’ domain b and Riemann domain d for $\mu$-band (left) and $\beta$-band (right) in the first and last 15 runs of 2019 and 2020 training periods. The box edges signify the 75th (top) and 25th (bottom) percentiles and the horizontal line the median of the corresponding distribution. The whiskers extend to the largest and smallest nonoutlier values. Single-run values are marked with filled circles. Statistically significant differences are shown with Tukey-Kramer post-hoc tests

Fig. 6

Within-class distance in channels’ and Riemann domains. a–c Evolution over races of the within-class distance in channels’ domain a and Riemann domain c computed in the $\mu$-band (8–12 Hz) and $\beta$-band (16–26 Hz). Their corresponding linear fits and Pearson correlation coefficients (significance tested with Student t test distribution) were evaluated for the two years (2019, 2020) separately. Vertical thin lines indicate the date of each racing session, while vertical thick black lines represent the dates of decoder update. The break of 1 year is marked by a vertical red line. b–d Boxplots of within-class distance in channels’ domain b and Riemann domain d for $\mu$-band (left) and $\beta$-band (right) in the first and last 15 runs of 2019 and 2020 training periods. The box edges signify the 75th (top) and 25th (bottom) percentiles and the horizontal line the median of the corresponding distribution. The whiskers extend to the largest and smallest nonoutlier values. Single-run values are marked with filled circles. Statistically significant differences are shown with Tukey-Kramer post-hoc tests