Neurotechnology for enhancing human operation of robotic and semi-autonomous systems

Abstract

Human operators of remote and semi-autonomous systems must have a high level of executive function to safely and efficiently conduct operations. These operators face unique cognitive challenges when monitoring and controlling robotic machines, such as vehicles, drones, and construction equipment. The development of safe and experienced human operators of remote machines requires structured training and credentialing programs. This review critically evaluates the potential for incorporating neurotechnology into remote systems operator training and work to enhance human-machine interactions, performance, and safety. Recent evidence demonstrating that different noninvasive neuromodulation and neurofeedback methods can improve critical executive functions such as attention, learning, memory, and cognitive control is reviewed. We further describe how these approaches can be used to improve training outcomes, as well as teleoperator vigilance and decision-making. We also describe how neuromodulation can help remote operators during complex or high-risk tasks by mitigating impulsive decision-making and cognitive errors. While our review advocates for incorporating neurotechnology into remote operator training programs, continued research is required to evaluate the how these approaches will impact industrial safety and workforce readiness.

Keywords: BCI; attention; cognition; human-machine; neuromodulation; neurotechnology; teleoperations.

FIGURE 1

Commercial and industrial utility of remotely piloted drones. The iconographic illustrations depict some example applications of sUAS/UAVs that are creating an impact on many different industries. The development and deployment of different types of sensors and edge computing methods has led to a growing adoption of drones and other remotely operated robotic systems for diverse industrial applications as illustrated. This broadening use of semi-autonomous drones and robotic systems in commercial, security, and defense applications means workforce and talent development efforts will also grow to include the establishment of specific training programs and certifications. Some training programs may incorporate modern neurotechnology methods to enhance remote operator cognition and performance.

FIGURE 2

Human interfaces for training and operation of drones and remotely controlled machines. (A) Modern aerial drones utilize different types of digital human control interfaces, such as joysticks with integrated screens (left) to heads-up displays with integrated motion controllers (middle). Some drone and remote vehicle interfaces implement analog transmission of control, video, and sensor data with directional RF antennas, displays, and joysticks (right). The displays and controllers in these systems convey important information to the operator from optical sensors, gyroscopes and accelerometers, environmental sensors, positioning systems, and many other types of sensors and actuators. (B) Simulator control systems can prepare remote operators for real-world operations. Modern simulator equipment may include joysticks and foot pedals (left) or operator’s chairs (middle) that can be connected to computers, displays, and virtual reality systems to replicate vehicle and environmental situations. These simulators have proven indispensable to the cognitive and sensorimotor training of operators for complex robotic tasks like remote excavation (right). The left and middle image were reproduced from reference (Burk et al., 2023) and the right image was reproduced from reference (Hiltunen et al., 2023).

FIGURE 3

Wearable electrical neuromodulation approaches for augmenting human-machine interactions. Several forms of noninvasive neuromodulation are wearable and can be easily used before, during, or after virtual training activities on simulators or used during human-machine operations of drones or other robotic systems. (A) Transcranial electrical stimulation including transcranial direct and alternating current stimulation methods involve the placement of electrodes at different locations on the scalp to modulate cognitive, sensory, and motor networks. (B) Transcutaneous cervical vagus nerve stimulation involves placement of surface electrode contacts along the side of the neck to modulate activity of cervical branches of the vagus. (C) Several different approaches to transcutaneous auricular vagus nerve stimulation (taVNS) have been developed. As shown on the far-right some taVNS methods use small, steel ball electrodes to stimulate the external ear (Nemos device, Cerbomed GmbH). This approach can produce high, local current densities resulting in discomfort or electrical biting and stinging sensations. Other taVNS methods and devices utilize metal clip electrodes as shown in the middle-left image. These clips mechanically couple the skin to a metal electrode using an electrolyte solution or gel. This approach creates a distracting pinching sensation and can produce electrical biting or prickling sensations. Other methods were developed using conductive hydrogel earbud electrodes (BRAIN Buds, IST) to produce uniform current distributions and enhanced user comfort during taVNS.

FIGURE 4

Noninvasive neuromodulation approaches for enhancing remote operator training and performance. The diagram illustrates how noninvasive neuromodulation methods may be used to enhance different aspects of human-machine operator performance by improving skill acquisition and retention, improving cognitive abilities, and improving operational safety.

FIGURE 5

Wearable transcranial focused ultrasound neuromodulation devices. Recent breakthroughs in human factors, mechanical, materials, biomedical, and electrical engineering have enabled the development of wearable devices for neuromodulation by low-intensity transcranial focused ultrasound (LIFU). The device shown (Attune Neuroscience, Inc.) utilizes LIFU transducers to target different regions of the human brain and integrates electroencephalography (EEG) sensors for measuring brain wave activity patterns from the prefrontal cortex. This type of wearable device for neuromodulation by LIFU opens the possibility of developing closed-loop applications for human-in-the-loop control of robotic and semi-autonomous systems. Figure adapted from Reference (Meads et al., 2024).

FIGURE 6

Brain-computer interface approaches for real-time optimization of remote pilot and driver performance. (A) Several types of biometric data including brain signals from electroencephalography (EEG), heart rate, and heart rate variability can be used to infer stress, attention, and fatigue in teleoperators and drivers. These biometric signals can used alone or in conjunction with signals and cues from human-machine interfaces and controllers as triggers or commands to elicit transcutaneous auricular vagus nerve stimulation (taVNS), transcranial electrical stimulation (tES), and transcranial focused ultrasound (tFUS) for enhancing the human operation of robotic machines and semi-autonomous vehicles. (B) The photographs illustrate a taVNS device integrated with a first-person view (FPV) display and headset receiving video feeds and telemetric data from a sUAS. This head-mounted taVNS approach is useful for delivering, real-time bilateral taVNS during FPV sUAS training and flight operations. (C) The photographs illustrate a subject wearing an EEG cap for monitoring attention and cognitive states while an eye tracking system is used to monitor visual attention and gaze during driving simulations. The subject is also wearing taVNS electrodes to responsively modulate attention and decision-making. Using EEG signals and data from the eye tracking system, taVNS can be triggered in a closed-loop manner that is responsive to an individual’s cognitive load, attention level, or stress to modulate operator neuroplasticity and cognition during training sessions in synthetic or virtual environments.