The Brain Is Faster than the Hand in Split-Second Intentions to Respond to an Impending Hazard: A Simulation of Neuroadaptive Automation to Speed Recovery to Perturbation in Flight Attitude

Abstract

The goal of this research is to test the potential for neuroadaptive automation to improve response speed to a hazardous event by using a brain-computer interface (BCI) to decode perceptual-motor intention. Seven participants underwent four experimental sessions while measuring brain activity with magnetoencephalograpy. The first three sessions were of a simple constrained task in which the participant was to pull back on the control stick to recover from a perturbation in attitude in one condition and to passively observe the perturbation in the other condition. The fourth session consisted of having to recover from a perturbation in attitude while piloting the plane through the Grand Canyon constantly maneuvering to track over the river below. Independent component analysis was used on the first two sessions to extract artifacts and find an event related component associated with the onset of the perturbation. These two sessions were used to train a decoder to classify trials in which the participant recovered from the perturbation (motor intention) vs. just passively viewing the perturbation. The BCI-decoder was tested on the third session of the same simple task and found to be able to significantly distinguish motor intention trials from passive viewing trials (mean = 69.8%). The same BCI-decoder was then used to test the fourth session on the complex task. The BCI-decoder significantly classified perturbation from no perturbation trials (73.3%) with a significant time savings of 72.3 ms (Original response time of 425.0-352.7 ms for BCI-decoder). The BCI-decoder model of the best subject was shown to generalize for both performance and time savings to the other subjects. The results of our off-line open loop simulation demonstrate that BCI based neuroadaptive automation has the potential to decode motor intention faster than manual control in response to a hazardous perturbation in flight attitude while ignoring ongoing motor and visual induced activity related to piloting the airplane.

Keywords: MEG; aviation; brain computer interface; brain machine interface; decoding; independent component analysis; neuroadaptive automation; neuroergonomics.

Figure 1

Outline of processing procedures for the implementation of the hypothesized neuroadaptive automation to speed recovery to perturbation in flight attitude. The goal of this system is to speed up response time for the aircraft to recover from a perturbation by decoding the motor intention of the pilot. In this way the pilot is always in control of the aircraft rather than relying on automation in which the pilot is out of the loop. It should be noted that all processing was done offline and that the online parts of the system were simulated. The processing times for each of the procedures if ran in real-time online are given. As we were carrying out an offline simulation to determine the feasibility of signal processing and the BCI-decoder performance during training and testing stages for implementation in a real-time neuroadaptive automation system the aircraft computer was not actually implemented in this study. The system is theoretically able to work in real time with only a 5–7.5 ms loss in time savings because the weights of the ICA and BCI-decoder are determined before hand and applied to the online system. The aircraft computer is a necessary part of the neuroadaptive automation system that receives commands from the BCI-decoder to implement the recovery maneuver (in this case upward elevator deflection). The aircraft computer can also send information to the BCI-Decoder that can signal the onset of potential perturbations to the airplane. This information can be used to reduce the occurrence of false-alarms made by the system (executing upward elevator deflection when there is no actual perturbation or motor intention to recover). The aircraft computer can use up to 120 ms (time of the processing window for the BCI-decoder) to determine the presence of a non-pilot initiated perturbation in attitude without causing a loss in the time savings afforded by the neuroadaptive automation system. ICA, Independent Component Anayalsis; BCI, Brain Computer Interface; LSPC, Least Squares Probabilistic Classification; UDP, Universal Datagram Protocol.

Figure 2

First person view the participant observes while carrying out the simple piloting task over the ocean (A,B) and the complex piloting task (C,D). The first panel for each task (A,C) shows a representative image of what the view may appear like prior to the perturbation. The second panel for each task (B,D) shows a representative image of what the view may appear like during the perturbation. Notice that in the simple piloting task over the ocean (A,B) the bank angle is always level, whereas, in the complex piloting task the bank angle is continuously changing based on the control stick inputs to maintain the goal of tracking the river (See Supplementary Videos 1–6).

Figure 3

Source localized activity for each participant (P01–P07 denotes participant identification number). (A) On the left the independent component analysis spatial filters for the MEG channels are shown for each participant. (B) The mean activation waveform for session one with the peak latency given in the upper corner for each participant. The blue boxes over the peak denote the three 40 ms windows the decoder was trained on. The mean response time is denoted by the gray line in the plot. The corresponding value is shown to the bottom right of this line for each participant. (C) The estimated current using variational Baysian multimodal encephalography VBMEG is shown rendered on the surface of the brain for each participant.

Figure 4

Source localized activity common to all participants. Activity is present in the left pre- central gyrus, the left post central gyrus, the right superior parietal lobule, the right primary visual cortex V1, and the right visual motion processing area V5.

Figure 5

The decoded response time (circles) plotted on the single trial activation waveforms (ranging from: red: large positive amplitude to blue: large negative amplitude) of the selected independent component of the simulated neuroadaptive automation on the complex piloting task for (A) the best participant (P01) and (B) the middle range participant in terms of classification performance (P03). Both perturbation absent trials (top of each plot) and perturbation present trials (bottom of each plot) are shown. The perturbation present trials are arranged in order of fastest manual response time (bottom) to the slowest (top). The manual response times are denoted by the thick white line for the perturbation present trials. The decoded response time, by the simulated neuroadaptive automation (BCI classifier), of each trial is denoted by a black circle. For perturbation present trials the black circles denote hits when their time is faster than the manual response time (white line). For perturbation absent trials the black circles denote false alarms. A red circle is shown over the original response time in the case when the simulated neuroadaptive automation failed to classify the trial as a hit (misses) or in which it was slower than the original response time (slow responses).