State-based decoding of hand and finger kinematics using neuronal ensemble and LFP activity during dexterous reach-to-grasp movements

Abstract

The performance of brain-machine interfaces (BMIs) that continuously control upper limb neuroprostheses may benefit from distinguishing periods of posture and movement so as to prevent inappropriate movement of the prosthesis. Few studies, however, have investigated how decoding behavioral states and detecting the transitions between posture and movement could be used autonomously to trigger a kinematic decoder. We recorded simultaneous neuronal ensemble and local field potential (LFP) activity from microelectrode arrays in primary motor cortex (M1) and dorsal (PMd) and ventral (PMv) premotor areas of two male rhesus monkeys performing a center-out reach-and-grasp task, while upper limb kinematics were tracked with a motion capture system with markers on the dorsal aspect of the forearm, hand, and fingers. A state decoder was trained to distinguish four behavioral states (baseline, reaction, movement, hold), while a kinematic decoder was trained to continuously decode hand end point position and 18 joint angles of the wrist and fingers. LFP amplitude most accurately predicted transition into the reaction (62%) and movement (73%) states, while spikes most accurately decoded arm, hand, and finger kinematics during movement. Using an LFP-based state decoder to trigger a spike-based kinematic decoder [r = 0.72, root mean squared error (RMSE) = 0.15] significantly improved decoding of reach-to-grasp movements from baseline to final hold, compared with either a spike-based state decoder combined with a spike-based kinematic decoder (r = 0.70, RMSE = 0.17) or a spike-based kinematic decoder alone (r = 0.67, RMSE = 0.17). Combining LFP-based state decoding with spike-based kinematic decoding may be a valuable step toward the realization of BMI control of a multifingered neuroprosthesis performing dexterous manipulation.

Keywords: brain-machine interface; movement decoding; neuroprosthetics; state decoding.

Fig. 1.

Experimental setup and task apparatus. A: subject was instructed to release a centrally located home object and reach toward, grasp, and manipulate 1 of 4 peripheral objects (sphere, mallet, pushbutton, pull handle) arranged in a planar circle. A set of blue LEDs cued which object to reach for, and green LEDs indicated when the object was manipulated. Upper limb kinematics were tracked with 30 optically reflective markers affixed to the monkey's right forearm, hand, and digits. B: drawings from digital video frames are shown adjacent to the reconstructed kinematics from the motion tracking system (stick figures). Grasping each object invoked a unique grasp conformation of the fingers and wrist. C: for monkey X, marker placement included 4 markers on the dorsal aspect of the forearm, 6 markers on the dorsal surface of the hand, 2 markers over the first metacarpal, 2 markers between the metacarpophalangeal (MCP) and proximal interphalangeal (PIP) joints for each digit, and 2 markers between the PIP and distal interphalangeal (DIP) joints for digits 2 through 5. For monkey Y, 5 markers were placed on the dorsal surface of the hand and the forearm. Distribution of kinematic data for monkey X is shown in Cartesian space (D; shown separately for x-, y-, and z-dimensions) and joint angle space (E; shown separately for joint angles of the wrist, thumb, and fingers).

Fig. 2.

Neural recording locations. Simultaneous spiking and local field potential (LFP) activity was recorded from multiple floating microarrays (FMAs) in primary motor cortex (M1) and dorsal (PMd) and ventral (PMv) premotor areas of 2 subjects (monkeys X and Y). Left: drawings traced from intraoperative photos show the location of implanted arrays in each monkey, relative to cortical sulci (C.S., central sulcus; A.S., arcuate sulcus; S.P.S. superior precentral sulcus). Only the arrays used in the present study are shown. Right: neural recording sites from electrodes of staggered (i.e., nonuniform) length are indicated by circles. Open circles identify spike recording sites, and filled circles identify simultaneous spike and LFP recording sites. Spikes were acquired from every electrode, while LFPs were acquired from every other electrode. Rectangular prisms outlined in gray outline indicate the approximate recording volume sampled by each FMA.

Fig. 3.

Neural and kinematic activity during reach and grasp. A: time-averaged traces of spiking activity, LFP amplitude, and LFP power for a representative single unit and LFP channel recorded from the same electrode, averaged across all correctly performed trials for 1 movement type in 1 session (monkey X: pushbutton in session X0918). Ten individual trials of spiking activity are shown as dot rasters directly below the histogram of spiking activity (black trace). Ten trials of LFP activity (gray traces) are shown directly below the average LFP amplitude (black trace). B: time-averaged traces of selected joint angles from the wrist (wristRot, wrist rotation) and fingers (midAbAd, middle abduction/adduction; pnkMCP, pinky metacarpal) as well as hand centroid position (handX, average of palm markers in x-dimension). Ten individual trials of kinematic data (gray traces) are shown overlaid with the average activity (black trace) for each degree of freedom (DoF). The average time of cue presentation (Cue), onset of movement (OM), start of static hold (SH), and final hold completion (FH) are indicated by symbols above each set of traces. Horizontal bars depict the behavioral states delimited by these event markers: baseline (base), period 300 ms prior to Cue; reaction (rxn), period from Cue to OM; movement (move), period from OM to SH; hold, period from SH to FH. For each DoF, the kinematic data were normalized to zero mean and unit standard deviation across all trials. All trials were aligned at SH.

Fig. 4.

Predicting behavioral states and kinematics. A: average decoding accuracy for prediction of 4 behavioral states with LFP power in different frequency bands, LFP amplitude, and spike firing rate. Top: decoding results averaged across all 4 states as a function of the number of spike recordings or LFP channels. Bottom: decoding results for each state separately for a fixed common number of spike recordings or LFP channels (n = 15). Horizontal dashed line indicates chance state decoding accuracy (25%). B: average correlation coefficients (CC) and root mean square error (RMSE) for prediction of arm, hand, and finger kinematics from Cue to SH with LFP power in different frequency bands, LFP amplitude, and spike firing rate. Top: decoding results averaged across all 18 joint angle and 3 end point DoFs as a function of the number of spike recordings or LFP channels. Bottom: decoding results grouped separately for hand end point, average joint angles of the wrist, and average joint angles of each digit (thumb, index, middle, ring, little), for a common fixed number of spike recordings or LFP channels (n = 55). Asterisks indicate chance levels of kinematic correlations. All results shown are averaged across both monkeys and all 4 sessions. Error bars indicate SE.

Fig. 5.

Reconstruction of arm, hand, and finger kinematics. Reconstruction of selected joint angles from the wrist, hand, and fingers using decoded output from either LFP amplitude (left) or spike recordings (right) from 1 session in each monkey (X1002, Y0211). Results are shown for 3 sample trials of reach-to-grasp movements to each of the 4 object types (mallet, sphere, pushbutton, pull handle). Correlation coefficients (r) are reported as mean across all 18 joint angles and 3 end point DoFs.

Fig. 6.

Decoding accuracy as a function of motor region. Mean state decoding accuracy (A) and kinematic correlation coefficients (B) using neural activity randomly sampled from only M1 arrays (monkey X: arrays F, G, H, I, J; monkey Y: arrays G, H, I, J), PMv/PMd arrays (monkey X: arrays C, D; monkey Y: arrays A, B, C, D), or all arrays. Results shown used a fixed common number of spike recordings or LFP channels (n = 15) and then were averaged across the 4 sessions, 2 from each monkey. Horizontal dashed line in A indicates chance state decoding accuracy (25%). Asterisks in B indicate chance kinematic correlations. Error bars indicate SE.

Fig. 7.

State transition latencies. A: state diagram illustrates transition from one behavioral state to next. To eliminate spurious state transitions, predicted transition times were determined as the time at which the state decoder correctly predicted the new state at least β times out of the previous τ decision points (see inset). B: mean state transition latencies using LFP amplitude and spike recordings randomly sampled from only M1 arrays, PMv/PMd arrays, or all arrays. State transition latencies are reported as the difference between the predicted transition times and actual transition times as indicated by behavioral markers for Cue (baseline-reaction), OM (reaction-movement), and SH (movement-hold). Results were averaged across only those trials where decoding accuracy for each state was at least 25%. Asterisks indicate significant differences between state transition latencies. Error bars indicate SE.

Fig. 8.

Combined state-based kinematic decoding. A: comparison of actual kinematics (blue) for a single DoF, wristFE, with predicted kinematics from a kinematic decoder using spike recordings (green) or a state-based kinematic decoder using LFP amplitude for state decoding and spikes for kinematic decoding (red). Results are shown for a sample trial from session X0918. B: average CC and RMSE for prediction of arm, hand, and finger kinematics from 300 ms prior to Cue to FH, from 1) only a spike-based kinematic decoder (blue); 2) a spike-based state decoder combined with a spike-based kinematic decoder (green); and 3) an LFP-based state decoder combined with a spike-based kinematic decoder (orange). Decoding results averaged across all 18 joint angle and 3 end point DoFs as a function of different numbers of randomly selected spike recordings for the kinematic decoder (x-axis). Error bars indicate SE.