Investigating an Integrated Sensor Fusion System for Mental Fatigue Assessment for Demanding Maritime Operations

Abstract

Human-related issues are currently the most significant factor in maritime causalities, especially in demanding operations that require coordination between two or more vessels and/or other maritime structures. Some of these human-related issues include incorrect, incomplete, or nonexistent following of procedures; lack of situational awareness; and physical or mental fatigue. Among these, mental fatigue is especially dangerous, due to its capacity to reduce reaction time, interfere in the decision-making process, and affect situational awareness. Mental fatigue is also especially hard to identify and quantify. Self-assessment of mental fatigue may not be reliable and few studies have assessed mental fatigue in maritime operations, especially in real time. In this work we propose an integrated sensor fusion system for mental fatigue assessment using physiological sensors and convolutional neural networks. We show, by using a simulated navigation experiment, how data from different sensors can be fused into a robust mental fatigue assessment tool, capable of achieving up to 100 % detection accuracy for single-subject classification. Additionally, the use of different sensors seems to favor the representation of the transition between mental fatigue states.

Keywords: deep learning; maritime operations; mental fatigue; physiological sensors.

Figure 1

Proposed concept for fatigue assessment.

Figure 2

Sensor setup. (a) Selected sensors and how to wear them. (b) MySignals Arduino shield and protective case. (c) Data acquisition application implemented in Java.

Figure 3

Proposed sensor fusion structure.

Figure 4

Raw data fusion scheme. The data from different sensors’ channels is segmented using a sliding window and concatenated as a 2D input for the CNN.

Figure 5

Description of the implemented CNN structure. The number between parenthesis below each convolutional layer represents the number of convolutional filters in that layer.

Figure 6

Labeling approach used to convert Karolisnka Sleepiness Scale (KSS) score to our proposed mental fatigue (MF) scale for CNN training.

Figure 7

Experimental setup on vessel simulator, at the Norsk Maritimt Kompetansesenter.

Figure 8

Reported KSS range of each test subject. For each subject, the lower limit shows the reported KSS level at the beginning of the experiment and the upper limit shows the reported KSS level at the end of the experiment.

Figure 9

MF assessment for different cases for Subject 1. The horizontal axis is presented in seconds. The vertical green dashed line marks the superior limit of the data used for training the non-fatigue condition. The vertical red dashed line marks the inferior limit of the data used for training the fatigue condition. Different cases present good agreement among themselves.

Figure 10

MF assessment for different cases for Subject 3. The horizontal axis is presented in seconds. The vertical green dashed line marks the superior limit of the data used for training the non-fatigue condition. The vertical red dashed line marks the inferior limit of the data used for training the fatigue condition. Case 3 produced a sharp transition between the fatigue and non-fatigue states. The addition of other sensors data helps with producing a smoother transition phase.

Figure 11

MF assessment for Case 3, for all available subjects. The horizontal axis is presented in seconds. The vertical green dashed line marks the superior limit of the data used for training the non-fatigue condition. The vertical red dashed line marks the inferior limit of the data used for training the fatigue condition. The EEG sensor presented the best classification performance, but transitions between different MF levels can be abrupt.

Figure 12

Projection of biggest activation of each CNN layer in the input domain for Subject 11 and KSS = 2.

Figure 13

Projection of biggest activation of each CNN layer in the input domain for Subject 11 and KSS = 4.