Interpretable Machine Learning Models for Three-Way Classification of Cognitive Workload Levels for Eye-Tracking Features

Abstract

The paper is focussed on the assessment of cognitive workload level using selected machine learning models. In the study, eye-tracking data were gathered from 29 healthy volunteers during examination with three versions of the computerised version of the digit symbol substitution test (DSST). Understanding cognitive workload is of great importance in analysing human mental fatigue and the performance of intellectual tasks. It is also essential in the context of explanation of the brain cognitive process. Eight three-class classification machine learning models were constructed and analysed. Furthermore, the technique of interpretable machine learning model was applied to obtain the measures of feature importance and its contribution to the brain cognitive functions. The measures allowed improving the quality of classification, simultaneously lowering the number of applied features to six or eight, depending on the model. Moreover, the applied method of explainable machine learning provided valuable insights into understanding the process accompanying various levels of cognitive workload. The main classification performance metrics, such as F1, recall, precision, accuracy, and the area under the Receiver operating characteristic curve (ROC AUC) were used in order to assess the quality of classification quantitatively. The best result obtained on the complete feature set was as high as 0.95 (F1); however, feature importance interpretation allowed increasing the result up to 0.97 with only seven of 20 features applied.

Keywords: cognitive workload; explainable machine learning; eyetracking signal; mutliclass classification.

Figure 1

The interface of the application. The highlighted area presents the current symbol to be matched.

Figure 2

The procedure of the experiment.

Figure 3

Feature selection experiment flowchart.

Figure 4

Separate feature importance weights.

Figure 5

F1 scores for various feature numbers. Logistic Regression.

Figure 6

F1 scores for various feature numbers. Linear Support Vector Machines (Linear SVM).

Figure 7

F1 scores for various feature numbers. Multilayer Perceptron (MLP).

Figure 8

The comparison of mean values from the three-part digit symbol substitution test (DSST) for selected features.