Classification of Drivers' Mental Workload Levels: Comparison of Machine Learning Methods Based on ECG and Infrared Thermal Signals

Abstract

Mental workload (MW) represents the amount of brain resources required to perform concurrent tasks. The evaluation of MW is of paramount importance for Advanced Driver-Assistance Systems, given its correlation with traffic accidents risk. In the present research, two cognitive tests (Digit Span Test-DST and Ray Auditory Verbal Learning Test-RAVLT) were administered to participants while driving in a simulated environment. The tests were chosen to investigate the drivers' response to predefined levels of cognitive load to categorize the classes of MW. Infrared (IR) thermal imaging concurrently with heart rate variability (HRV) were used to obtain features related to the psychophysiology of the subjects, in order to feed machine learning (ML) classifiers. Six categories of models have been compared basing on unimodal IR/unimodal HRV/multimodal IR + HRV features. The best classifier performances were reached by the multimodal IR + HRV features-based classifiers (DST: accuracy = 73.1%, sensitivity = 0.71, specificity = 0.69; RAVLT: accuracy = 75.0%, average sensitivity = 0.75, average specificity = 0.87). The unimodal IR features based classifiers revealed high performances as well (DST: accuracy = 73.1%, sensitivity = 0.73, specificity = 0.73; RAVLT: accuracy = 71.1%, average sensitivity = 0.71, average specificity = 0.85). These results demonstrated the possibility to assess drivers' MW levels with high accuracy, also using a completely non-contact and non-invasive technique alone, representing a key advancement with respect to the state of the art in traffic accident prevention.

Keywords: ADAS; automotive ergonomics; driver monitoring; infrared imaging; mental workload.

Figure A1

Thermal signals extracted from nosetip (in blue) and glabella (in orange) ROIs over the experimental phases.

Figure A2

Exemplificative ECG signal. Example of R peaks and RR intervals are highlighted.

Figure 1

Experimental setting: (a) static driver simulator; (b) screenshot of the driving simulation software (i.e., City Car Driving, Home Edition software-version 1.5, Forward Development, Ltd., Verona (WI), USA [37]).

Figure 2

Pipeline of the experimental protocol.

Figure 3

Imaging system device settings: (a) position of the imaging system in the driving simulator; (b) detail of the imaging system device (visible and thermal camera horizontally aligned and held together by means of a 3d-printed support).

Figure 4

Processing of thermal and visible videos. (a) Software interface for the acquisition and processing of visible and thermal IR videos. (b) Thermal image with ROI drawn in red colors (Nosetip and Glabella); (c) thermal signal extracted from the two ROIs during the experimental phases. The values are obtained subtracting the mean value of the signals during the baseline phase.

Figure 5

Whiskers plot of the participants’ scores in DST (a) and RAVLT (b). Outliers are represented with red crosses.

Figure 6

Thermal features relative to Nosetip ROI extracted during DST (* p < 0.05; ** p < 0.01). Outliers are represented with red crosses. The titles of the single plots refer to abbreviations of features described in Section 3.3.

Figure 7

Thermal features relative to Glabella ROI extracted during DST (** p < 0.01). Outliers are represented with red crosses. The titles of the single plots refer to abbreviations of features described in Section 3.3.

Figure 8

HRV features extracted during DST. Outliers are represented with red crosses. The titles of the single plots refer to abbreviations of features described in Section 3.3.

Figure 9

Thermal features relative to Nosetip ROI extracted during RAVLT (* p < 0.05; ** p < 0.01). Outliers are represented with red crosses. The titles of the single plots refer to abbreviations of features described in Section 3.3.

Figure 10

Thermal features relative to Glabella ROI extracted during RAVLT (* p < 0.05; ** p < 0.01). Outliers are represented with red crosses. The titles of the single plots refer to abbreviations of features described in Section 3.3.

Figure 11

HRV features extracted during RAVLT (* p < 0.05; ** p < 0.01). Outliers are represented with red crosses. The titles of the single plots refer to abbreviations of features described in Section 3.3.

Figure 12

Scheme of classification adopted in the present work: (a) scheme of the two-level classification model for DST; (b) scheme of the three-level classification model for RAVLT.

Figure 13

Performances of the SVM classifiers for DST: (a) ROC curve for unimodal IR features-based classifier; (b) confusion matrix unimodal IR features-based classifier; (c) ROC curve for multimodal IR + HRV features-based classifier; (d) confusion matrix multimodal IR + HRV features-based classifier.

Figure 14

Performances of the Ensemble bagged trees for unimodal IR features-based classifiers for RAVLT: (a) ROC curve for the classifier of class ImmR (i.e., class 0) vs. cumulative class (DelR + Rec) (i.e., class 1 + 2); (b) ROC curve for the classifier of class DelR (i.e., class 1) vs. cumulative class (ImmR + Rec) (i.e., class 0 + 2); (c) ROC curve for the classifier of class Rec (i.e., class 2) vs. cumulative class (ImmR + DelR) (i.e., class 0 + 1); (d) confusion matrix for the unimodal IR features-based classifier.

Figure 15

Performances of the Ensemble bagged trees for multimodal IR + hrv features-based classifiers for RAVLT: (a) ROC curve for the classifier of class ImmR (i.e., class 0) vs. cumulative class (DelR + Rec) (i.e., class 1 + 2); (b) ROC curve for the classifier of class DelR (i.e., class 1) vs. cumulative class (ImmR + Rec) (i.e., class 0 + 2); (c) ROC curve for the classifier of class Rec (i.e., class 2) vs. cumulative class (ImmR + DelR) (i.e., class 0 + 1); (d) confusion matrix for the multimodal IR features-based classifier.