Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload

Abstract

While studies exist that compare different physiological variables with respect to their association with mental workload, it is still largely unclear which variables supply the best information about momentary workload of an individual and what is the benefit of combining them. We investigated workload using the n-back task, controlling for body movements and visual input. We recorded EEG, skin conductance, respiration, ECG, pupil size and eye blinks of 14 subjects. Various variables were extracted from these recordings and used as features in individually tuned classification models. Online classification was simulated by using the first part of the data as training set and the last part of the data for testing the models. The results indicate that EEG performs best, followed by eye related measures and peripheral physiology. Combining variables from different sensors did not significantly improve workload assessment over the best performing sensor alone. Best classification accuracy, a little over 90%, was reached for distinguishing between high and low workload on the basis of 2 min segments of EEG and eye related variables. A similar and not significantly different performance of 86% was reached using only EEG from single electrode location Pz.

Keywords: ECG; EEG; classification; combination; eye; physiology; skin conductance; workload.

Figure 1

Classification performance (2- vs. 0-back, 120 s) for separate features as resulting from SVM and elastic net classification models. The horizontal lines indicate chance level (bottom), 0.05 significance level (middle) and 0.01 significance level (top). Different shades in the background indicate the three different sensor groups EEG, Physiology and Eye. Error bars indicate estimates of the variance (s.e.m.s) based on the binomial distribution.

Figure 2

Classification performance (2- vs. 0-back, 120 s) for separate sensors. Conventions as in Figure 1. For comparison, performance of the best performing feature for each of sensor is depicted.

Figure 3

Classification performance for separate and combined sensor groups for SVM, elastic net and a model that combines the outputs from different single feature models (“decision level”). (A) performance in the default condition (2- vs. 0-back, 120 s of data, (B) for comparing 2- vs. 0-back over 30 s of data, (C) for comparing 2- vs. 1-back (120 s of data), (D) for comparing 1- vs. 0-back (120 s of data).

Figure 4

Classification performance in the default condition (2- vs. 0-back, 120 s) for the different sensor types and using all available input. The striped bars show the effect of adding the time feature. Conventions as in Figure 1.