Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface

Abstract

Brain-computer interfaces (BCIs) allow a user to interact with a computer system using thought. However, only recently have devices capable of providing sophisticated multi-dimensional control been achieved non-invasively. A major goal for non-invasive BCI systems has been to provide continuous, intuitive, and accurate control, while retaining a high level of user autonomy. By employing electroencephalography (EEG) to record and decode sensorimotor rhythms (SMRs) induced from motor imaginations, a consistent, user-specific control signal may be characterized. Utilizing a novel method of interactive and continuous control, we trained three normal subjects to modulate their SMRs to achieve three-dimensional movement of a virtual helicopter that is fast, accurate, and continuous. In this system, the virtual helicopter's forward-backward translation and elevation controls were actuated through the modulation of sensorimotor rhythms that were converted to forces applied to the virtual helicopter at every simulation time step, and the helicopter's angle of left or right rotation was linearly mapped, with higher resolution, from sensorimotor rhythms associated with other motor imaginations. These different resolutions of control allow for interplay between general intent actuation and fine control as is seen in the gross and fine movements of the arm and hand. Subjects controlled the helicopter with the goal of flying through rings (targets) randomly positioned and oriented in a three-dimensional space. The subjects flew through rings continuously, acquiring as many as 11 consecutive rings within a five-minute period. In total, the study group successfully acquired over 85% of presented targets. These results affirm the effective, three-dimensional control of our motor imagery based BCI system, and suggest its potential applications in biological navigation, neuroprosthetics, and other applications.

Figure 1. A diagrammatic representation of the presented BCI system.

Using specifically trained motor imaginations learned in single dimensional cursor tasks, subjects control the three-dimensional movement of a virtual helicopter. Raw EEG is temporally and spatially filtered to produce individualized control signal components. These components are weighted and digitized in a subject specific manner and output to influence control in the virtual world.

Figure 2. Three-dimensional helicopter control arrangements are shown in perspective (a), side (b) and top (c) views.

Users have independent control of forward, backward, up, down, and left and right rotation about the helicopter's z-axis. To go forward or back, subjects imagine moving or resting both hands respectively. To rotate the helicopter left or right, subjects imagine moving either the left or right hand respectively. Subjects imagine moving the tongue to raise the helicopter and moving the feet to lower it. Each control can be independently adjusted in strength according to user preference.

Figure 3. Speed, accuracy, and continuity of control are depicted in a characteristic run performed by Subject 1 (Session 5, Trial 9, Targets 4–7).

Only one target was visible to the subject at any given time. The presentation of a new ring occurred 1.5 seconds after a ring hit. The helicopter's position upon ring presentation is represented by a larger coloured sphere. Smaller coloured spheres represent the position of the helicopter sampled every 30 milliseconds. The subject started with the blue ring, and progressed through red, green, and yellow rings as illustrated by the colour bar on the bottom of the figure. The overall duration of this continuous portion of the run was 69.53 seconds.

Figure 4. Time-frequency analysis shows averaged power distributions across time and frequency for representative control electrodes during segments of single direction control for subject number 3.

Electrode C3 is on the left scalp hemisphere and electrode C4 is on the right. At time 0, the subject moved the helicopter in primarily one direction for .5 s. When a right turn is made, C3 shows ERD and C4 shows ERS. The opposite is true for a left turn. When both hands are imagined, both electrodes show periods of desynchronization, while the rest state results in both electrodes exhibiting synchronization. These changes in the time-frequency profiles may be leveraged to control two-dimensions of movement with only hand imaginations and volitional rest.

Figure 5. Performance quality metrics.

(a) Percent valid correct (PVC) is the ratio of total hits to total non-invalid attempts during each experimental session. (b) Percent total correct (PTC) includes invalid attempts in the calculation. (c) The average number of rings obtained per reset (ARR) is a metric of control continuity. (d) The average ring acquisition velocity (ARAV) is the average of the net distance travelled by the helicopter from ring presentation to acquisition divided by the time required to cover the distance. ARAV serves as a control speed metric.

Figure 6. Correlation coefficients between (a) LR and FB (b) LR and UD, and (c) FB and UD control signals at zero lag.

Calculations were made for each five-minute session.