A Machine Learning Approach to Classifying EEG Data Collected with or without Haptic Feedback during a Simulated Drilling Task

Abstract

Artificial Intelligence (AI), computer simulations, and virtual reality (VR) are increasingly becoming accessible tools that can be leveraged to implement training protocols and educational resources. Typical assessment tools related to sensory and neural processing associated with task performance in virtual environments often rely on self-reported surveys, unlike electroencephalography (EEG), which is often used to compare the effects of different types of sensory feedback (e.g., auditory, visual, and haptic) in simulation environments in an objective manner. However, it can be challenging to know which aspects of the EEG signal represent the impact of different types of sensory feedback on neural processing. Machine learning approaches offer a promising direction for identifying EEG signal features that differentiate the impact of different types of sensory feedback during simulation training. For the current study, machine learning techniques were applied to differentiate neural circuitry associated with haptic and non-haptic feedback in a simulated drilling task. Nine EEG channels were selected and analyzed, extracting different time-domain, frequency-domain, and nonlinear features, where 360 features were tested (40 features per channel). A feature selection stage identified the most relevant features, including the Hurst exponent of 13-21 Hz, kurtosis of 21-30 Hz, power spectral density of 21-30 Hz, variance of 21-30 Hz, and spectral entropy of 13-21 Hz. Using those five features, trials with haptic feedback were correctly identified from those without haptic feedback with an accuracy exceeding 90%, increasing to 99% when using 10 features. These results show promise for the future application of machine learning approaches to predict the impact of haptic feedback on neural processing during VR protocols involving drilling tasks, which can inform future applications of VR and simulation for occupational skill acquisition.

Keywords: electroencephalography (EEG); haptic feedback; machine learning; simulations.

Figure 1

Diagram of the experimental flow for application of the machine learning algorithms to the EEG data.

Figure 2

Novint Falcon device covered by a 3D printed drill shape for drilling simulation (similar figure appears in Grant, 2019).

Figure 3

Simulation scenes and experimental protocol for data collection. (A) Side view was only observable during the familiarization trials. (B) The front view was displayed during all experimental trials. (C) Experimental design. (Panels A and B are screen shots of drill with similar figures in https://www.mdpi.com/2076-3425/10/1/21, accessed on 20 August 2024).

Figure 4

Score of the top 20 most relevant features according to MRMR. Maximum score of 100.

Figure 5

Topographic map of the most relevant features and frequency bands. Colors close to red represent high mean values (close to 1), and colors close to dark blue represent low mean values (close to 0).