Combined Action Observation and Motor Imagery Neurofeedback for Modulation of Brain Activity

Abstract

Motor imagery (MI) and action observation have proven to be efficacious adjuncts to traditional physiotherapy for enhancing motor recovery following stroke. Recently, researchers have used a combined approach called imagined imitation (II), where an individual watches a motor task being performed, while simultaneously imagining they are performing the movement. While neurofeedback (NFB) has been used extensively with MI to improve patients' ability to modulate sensorimotor activity and enhance motor recovery, the effectiveness of using NFB with II to modulate brain activity is unknown. This project tested the ability of participants to modulate sensorimotor activity during electroencephalography-based II-NFB of a complex, multi-part unilateral handshake, and whether this ability transferred to a subsequent bout of MI. Moreover, given the goal of translating findings from NFB research into practical applications, such as rehabilitation, the II-NFB system was designed with several user interface and user experience features, in an attempt to both drive user engagement and match the level of challenge to the abilities of the subjects. In particular, at easy difficulty levels the II-NFB system incentivized contralateral sensorimotor up-regulation (via event related desynchronization of the mu rhythm), while at higher difficulty levels the II-NFB system incentivized sensorimotor lateralization (i.e., both contralateral up-regulation and ipsilateral down-regulation). Thirty-two subjects, receiving real or sham NFB attended four sessions where they engaged in II-NFB training and subsequent MI. Results showed the NFB group demonstrated more bilateral sensorimotor activity during sessions 2-4 during II-NFB and subsequent MI, indicating mixed success for the implementation of this particular II-NFB system. Here we discuss our findings in the context of the design features included in the II-NFB system, highlighting limitations that should be considered in future designs.

Keywords: action observation; brain computer interface; electroencephalography; motor imagery; neurofeedback.

Figure 1

Illustration of the complex handshake video used in the II-NFB system.

Figure 2

Experimental timeline for all 4 sessions. ^** = junction where the subject was asked to take as much time as they would like to rest and prepare for the next block.

Figure 3

Example of the interaction between NFB score and the video's color during II-NFB (with a new NFB score calculated every 500 ms). When the NFB score exceeds the lower or upper bounds of the threshold (i.e., the definition of the win and lose state at the current difficulty level), the Video Score decreases or increases by 1 (unless already at 1, then it remains at 1), and the Video Score will then not change for the next 2 s, to ensure that changes in the color gradient do not happen so quickly as to become imperceptible.

Figure 4

Mean log2 mu rhythm ERD/S values from both the left (A) and right (B) hemisphere during the NFB task, and left (C) and right (D) hemisphere during the MI task for the NFB (black circles) and Sham (red squares) groups. The log2 of the ERD/S values in the mu rhythm have an inverse relationship to sensorimotor activity. Bars represent standard deviation.

Figure 5

Decision tree for the CForest predictive model of the contralateral (left) EEG sensors during the NFB task. The data is partitioned according to split of the independent variable (either session or group) that garners the most significant effect. Where no split garners an effect of p < 0.05 (Bonferroni corrected) no split is shown. Red squares represent the partition with significantly lower ERD/S values (i.e., less sensorimotor activity) while blue squares represent the corresponding partition with higher ERD/S values. Split A: effect of session, with sessions 2–4 showing more sensorimotor activity than session 1. Split B: effect of group, with sham showing more sensorimotor activity than NFB during session 1. Split C: effect of group, with NFB showing more sensorimotor activity than sham during sessions 2–4. Split D: effect of session, with sessions 2 and 4 showing more sensorimotor activity than session 3. Split E: effect of session, with session 3 showing more sensorimotor activity than sessions 2 and 4. Split F: effect of session, with session 4 showing more sensorimotor activity than session 2. Split G: effect of session, with session 2 showing more sensorimotor activity than session 4.

Figure 6

Decision tree for the CForest predictive model of the ipsilateral (right) EEG sensors during the NFB task. The data is partitioned according to split of the independent variable (either session or group) that garners the most significant effect. Where no split garners an effect of p < 0.05 (Bonferroni corrected) no split is shown. Red squares represent the partition with significantly lower ERD/S values (i.e., less sensorimotor activity) while blue squares represent the corresponding partition with higher ERD/S values. Split A: effect of group, with NFB showing more sensorimotor activity than sham. Split B: effect of session, with sessions 1–2 showing more sensorimotor activity than sessions 3–4 in the sham group. Split C: effect of session, with sessions 2–4 showing more sensorimotor activity than session 1 in the NFB group. Split D: effect of session, with session 4 showing more sensorimotor activity than session 3. Split E: effect of session, with session 1 showing more sensorimotor activity than session 2. Split F: effect of session, with session 4 showing more sensorimotor activity than sessions 2–3. Split G: effect of session, with session 3 showing more sensorimotor activity than session 2.

Figure 7

Decision tree for the CForest predictive model of the contralateral (left) EEG sensors during the MI task. The data is partitioned according to split of the independent variable (either session or group) that garners the most significant effect. Where no split garners an effect of p < 0.05 (Bonferroni corrected) no split is shown. Red squares represent the partition with significantly lower ERD/S values (i.e., less sensorimotor activity) while blue squares represent the corresponding partition with higher ERD/S values. Split A: effect of session, with sessions 2–4 showing more sensorimotor activity than session 1. Split B: effect of group, with sham showing more sensorimotor activity than NFB during session 1. Split C: effect of group, with NFB showing more sensorimotor activity than sham during sessions 2–4. Split D: effect of session, with sessions 2 and 3 showing more sensorimotor activity than session 4. Split E: effect of session, with sessions 3–4 showing more sensorimotor activity than sessions 2 and 4. Split F: effect of session, with session 2 showing more sensorimotor activity than session 3. Split G: effect of session, with session 3 showing more sensorimotor activity than session 4.

Figure 8

Average difficulty level achieved by subjects in both the NFB and Sham groups. Note that there were two difficulty levels below 1 (i.e., 0 and−1). Error bars represent standard deviations.