Decoding three-dimensional reaching movements using electrocorticographic signals in humans

Abstract

Objective: Electrocorticography (ECoG) signals have emerged as a potential control signal for brain-computer interface (BCI) applications due to balancing signal quality and implant invasiveness. While there have been numerous demonstrations in which ECoG signals were used to decode motor movements and to develop BCI systems, the extent of information that can be decoded has been uncertain. Therefore, we sought to determine if ECoG signals could be used to decode kinematics (speed, velocity, and position) of arm movements in 3D space.

Approach: To investigate this, we designed a 3D center-out reaching task that was performed by five epileptic patients undergoing temporary placement of ECoG arrays. We used the ECoG signals within a hierarchical partial-least squares (PLS) regression model to perform offline prediction of hand speed, velocity, and position.

Main results: The hierarchical PLS regression model enabled us to predict hand speed, velocity, and position during 3D reaching movements from held-out test sets with accuracies above chance in each patient with mean correlation coefficients between 0.31 and 0.80 for speed, 0.27 and 0.54 for velocity, and 0.22 and 0.57 for position. While beta band power changes were the most significant features within the model used to classify movement and rest, the local motor potential and high gamma band power changes, were the most important features in the prediction of kinematic parameters.

Significance: We believe that this study represents the first demonstration that truly three-dimensional movements can be predicted from ECoG recordings in human patients. Furthermore, this prediction underscores the potential to develop BCI systems with multiple degrees of freedom in human patients using ECoG.

Figure 1. Behavioral task and electrode implants

A. A photograph of a typical ECoG implant is shown. The electrodes were implanted beneath the dura as part of an 8x8 grid, 1x4 strips, 1x6 strips, or 1x8 strips. Electrodes had an exposed diameter of 2.3mm and an inter-electrode spacing of 1 cm. B. Electrode locations for each patient were mapped onto an atlas brain, allowing for comparison of ECoG activity by cortical locations across patients. Electrode locations were based solely upon each patient’s clinical needs. C. The photograph displays the apparatus used for the center-out reaching task. A cube with 50 cm sides was placed in front of the patient. Target locations and reward feedback were provided with LED lights placed at the 8 corner targets and center target. D. Each trial began with a 1 second hold-A period in which the subject held their hand at the central target. A 2 second planning delay was used during which time the subject was cued to the target of the reach and instructed to plan but not initiate a reaching movement to the appropriate target. After the movement “go” cue, subjects initiated a reach to the target. A successful trial ended with completion of a hold-B period in which subjects held their hand at the exterior target location.

Figure 2. Machine-learning strategy

To predict 3D kinematics, a hierarchical PLS regression model was used. After feature extraction, a logistic regression model classified time windows as either movement or rest. Two PLS regression models were trained, one relating ECoG features and kinematics during movement periods, and a second PLS model relating ECoG features and kinematics during non-movement periods. The final model output was generated from the outputs of the two PLS models by using the logistic regression output to switch between them.

Figure 3. Exemplar classification of movement and rest periods

Exemplar predictions of movement classes were generated using consecutive trials within a single held-out test set from Patient 5. Black traces show the actual hand speed that was used to generate true movement class labels shown in blue. Predicted movement class labels are shown in red and correspond well with the actual labels.

Figure 4. Movement kinematics are predicted with accuracies better than chance

Distributions show the correlation coefficients (Pearson’s r) between predicted and actual movement kinematics generated using the hierarchical PLS regression model. Distributions were generated from test sets using the actual movement class labels to switch between the PLS regression models. Boxes represent the median, 25^th, and 75^th percentiles and whiskers extend to the most extreme data points with outliers greater than 2.7 standard deviations marked with a +. * - Distributions with significantly higher correlations than the surrogate model generated by shuffling the temporal relationship between ECoG and kinematics, † - Distributions with significantly higher correlations than the surrogate model generated by shuffling feature and channel weights.

Figure 5. Exemplar kinematic predictions

Full model predictions were generated using the predicted movement classes from the logistic regression to switch between PLS model outputs. Actual kinematic traces are shown in blue and predicted traces are shown in red. Predictions in each plot were generated from consecutive trials from a single test set from patient 5.

Figure 6. Average predicted trajectories

Average actual and predicted trajectories from Patient 5 are shown for each of the 8 targets. Predicted trajectories were generated by concatenating predicted velocity vectors obtained from the full model and were then averaged for each target. Correlation coefficients (Pearson’s r) between the actual and predicted movement trajectories for the 8 targets range from 0.86 to 0.99.

Figure 7. Logistic regression model weights for movement and rest classification

Average logistic regression model weights are shown for selected frequencies. Beta band power and LMP amplitude in primary sensorimotor cortices have the strongest prediction weights across patients with decreases in beta band power and LMP amplitude associated with movement periods. Regions shown in dark grey had no electrode coverage.

Figure 8. Topography of PLS prediction weights

Absolute values for movement-class PLS regression weights and activation patterns compared across cortical locations for all patients. The color scale represents the normalized absolute value of prediction weights and activation patterns by cortical location. While prediction weights are spread over diffuse regions of cortex, activation patterns show that kinematics are encoded most strongly in sensorimotor areas. Regions shown in dark grey represent regions with no electrode coverage.

Figure 9. Importance of ECoG feature types for kinematic prediction

Distributions represent the maximum absolute movement-class PLS prediction weights and activation patterns for each feature and the percentage of the highest 1% of weights from each feature across patients and electrodes. LMP and high gamma band features have the largest prediction weights, while activation patterns demonstrate that LMP, mu, beta, and high gamma bands all encode kinematics.

Figure 10. Exemplar Asynchronous Prediction

Asynchronous predictions were made by training a model using all time points (including movement trials and ITI periods). The plots show 50 seconds of consecutive actual and predicted kinematics from a single test fold, demonstrating that the method presented has the potential to be applied in real-time BCI applications.