Reinforcement learning of self-regulated β-oscillations for motor restoration in chronic stroke

Abstract

Neurofeedback training of Motor imagery (MI)-related brain-states with brain-computer/brain-machine interfaces (BCI/BMI) is currently being explored as an experimental intervention prior to standard physiotherapy to improve the motor outcome of stroke rehabilitation. The use of BCI/BMI technology increases the adherence to MI training more efficiently than interventions with sham or no feedback. Moreover, pilot studies suggest that such a priming intervention before physiotherapy might-like some brain stimulation techniques-increase the responsiveness of the brain to the subsequent physiotherapy, thereby improving the general clinical outcome. However, there is little evidence up to now that these BCI/BMI-based interventions have achieved operate conditioning of specific brain states that facilitate task-specific functional gains beyond the practice of primed physiotherapy. In this context, we argue that BCI/BMI technology provides a valuable neurofeedback tool for rehabilitation but needs to aim at physiological features relevant for the targeted behavioral gain. Moreover, this therapeutic intervention has to be informed by concepts of reinforcement learning to develop its full potential. Such a refined neurofeedback approach would need to address the following issues: (1) Defining a physiological feedback target specific to the intended behavioral gain, e.g., β-band oscillations for cortico-muscular communication. This targeted brain state could well be different from the brain state optimal for the neurofeedback task, e.g., α-band oscillations for differentiating MI from rest; (2) Selecting a BCI/BMI classification and thresholding approach on the basis of learning principles, i.e., balancing challenge and reward of the neurofeedback task instead of maximizing the classification accuracy of the difficulty level device; and (3) Adjusting the difficulty level in the course of the training period to account for the cognitive load and the learning experience of the participant. Here, we propose a comprehensive neurofeedback strategy for motor restoration after stroke that addresses these aspects, and provide evidence for the feasibility of the suggested approach by demonstrating that dynamic threshold adaptation based on reinforcement learning may lead to frequency-specific operant conditioning of β-band oscillations paralleled by task-specific motor improvement; a proposal that requires investigation in a larger cohort of stroke patients.

Keywords: beta oscillations; brain-computer interface; brain-machine interface; brain-robot interface; functional restoration; hand function; reinforcement learning; stroke.

Figure 1

Brain-machine interface (BMI) training environment presented with a healthy subject. Motor imagery (MI)-related modulation of oscillatory activity is detected by electroencephalogram, EEG (1), amplified (2) and processed in a BCI2000-based control system (3) to operate a commercially available electromechanical hand orthosis (4).

Figure 2

Time course of BMI session. Each BMI session lasted approximately 30 min and consisted of fifteen runs separated by short breaks. Every run consisted of 11 trials, each of which lasted 16 s. Each trial started with a rest period (2 s) while subjects were instructed to prepare for MI following an auditory cue (2 s preparation phase), and to imagine the respective reaching movement following a “start” cue (6 s MI phase), which was followed by a “rest” cue (6 s rest phase).

Figure 3

(A) Off-line re-analysis of the BMI data in the first session for different thresholds θ providing the true-positive rates (TPR), true-negative rates (TNR), classification accuracy (CA), correct response rate (CRR) and the zone of adjusted threshold (ZAT). ZAT peaks at θ = 1.2 corresponding to TPR = 20.9%, FPR = 9.9% and CRR = 72.2%. Hence, θ = 1.2 was selected for the succeeding BMI session. (B) Shows BMI thresholds in the course of training. (C) Exemplary off-line analysis after session 4, 5 and 6 illustrating on the evolution of BMI performance.

Figure 4

(A) Mean time-frequency plot of sensorimotor feedback electrodes (FC4, C4 and CP4) during a BMI training session showing the event-related spectral perturbations (ERSPs). (B) Evolution of β-ERD in the course of 20 feedback sessions. (C) Evolution of BMI control in the course of the training period.

Figure 5

The topoplot indicates the cortical distribution of β-ERD during the first (A) and final (B) training week. The three feedback electrodes are shown as black dots. (C) Final ERSP changes for different frequency bands derived from the exponential fit to the ERSP learning curves at the end of training. Data represents mean ± 95%-confidence interval. (D) Fugl-Meyer assessment scores for arm, wrist and fingers prior to and subsequent to the training period.