Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Feb 9;11(2):210.
doi: 10.3390/brainsci11020210.

Interpretable Machine Learning Models for Three-Way Classification of Cognitive Workload Levels for Eye-Tracking Features

Monika Kaczorowska  1 , Małgorzata Plechawska-Wójcik  1 , Mikhail Tokovarov  1
Affiliations

Interpretable Machine Learning Models for Three-Way Classification of Cognitive Workload Levels for Eye-Tracking Features

Monika Kaczorowska et al. Brain Sci. .

Abstract

The paper is focussed on the assessment of cognitive workload level using selected machine learning models. In the study, eye-tracking data were gathered from 29 healthy volunteers during examination with three versions of the computerised version of the digit symbol substitution test (DSST). Understanding cognitive workload is of great importance in analysing human mental fatigue and the performance of intellectual tasks. It is also essential in the context of explanation of the brain cognitive process. Eight three-class classification machine learning models were constructed and analysed. Furthermore, the technique of interpretable machine learning model was applied to obtain the measures of feature importance and its contribution to the brain cognitive functions. The measures allowed improving the quality of classification, simultaneously lowering the number of applied features to six or eight, depending on the model. Moreover, the applied method of explainable machine learning provided valuable insights into understanding the process accompanying various levels of cognitive workload. The main classification performance metrics, such as F1, recall, precision, accuracy, and the area under the Receiver operating characteristic curve (ROC AUC) were used in order to assess the quality of classification quantitatively. The best result obtained on the complete feature set was as high as 0.95 (F1); however, feature importance interpretation allowed increasing the result up to 0.97 with only seven of 20 features applied.

Keywords: cognitive workload; explainable machine learning; eyetracking signal; mutliclass classification.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
The interface of the application. The highlighted area presents the current symbol to be matched.
Figure 2
Figure 2
The procedure of the experiment.
Figure 3
Figure 3
Feature selection experiment flowchart.
Figure 4
Figure 4
Separate feature importance weights.
Figure 5
Figure 5
F1 scores for various feature numbers. Logistic Regression.
Figure 6
Figure 6
F1 scores for various feature numbers. Linear Support Vector Machines (Linear SVM).
Figure 7
Figure 7
F1 scores for various feature numbers. Multilayer Perceptron (MLP).
Figure 8
Figure 8
The comparison of mean values from the three-part digit symbol substitution test (DSST) for selected features.
See this image and copyright information in PMC

References

    1. Gevins A., Smith M.E., McEvoy L., Yu D. High-resolution EEG mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice. Cereb. Cortex. 1997;7:374–385. doi: 10.1093/cercor/7.4.374. - DOI - PubMed
    1. Qi P., Ru H., Sun Y., Zhang X., Zhou T., Tian Y., Thakor N., Bezerianos A., Li J., Sun Y. Neural Mechanisms of Mental Fatigue Revisited: New Insights from the Brain Connectome. Engineering. 2019;5:276–286. doi: 10.1016/j.eng.2018.11.025. - DOI
    1. Gavelin H.M., Neely A.S., Dunås T., Eskilsson T., Järvholm L.S., Boraxbekk C.-J. Mental fatigue in stress-related exhaustion disorder: Structural brain correlates, clinical characteristics and relations with cognitive functioning. NeuroImage Clin. 2020;27:102337. doi: 10.1016/j.nicl.2020.102337. - DOI - PMC - PubMed
    1. Grier R.A., Warm J.S., Dember W.N., Matthews G., Galinsky T.L., Szalma J.L., Parasuraman R. The Vigilance Decrement Reflects Limitations in Effortful Attention, Not Mindlessness. Hum. Factors J. Hum. Factors Ergon. Soc. 2003;45:349–359. doi: 10.1518/hfes.45.3.349.27253. - DOI - PubMed
    1. Van der Linden D., Eling P. Mental fatigue disturbs local processing more than global processing. Psychol. Research. 2006;70:395–402. doi: 10.1007/s00426-005-0228-7. - DOI - PubMed