A New Perspective for the Training Assessment: Machine Learning-Based Neurometric for Augmented User's Evaluation

Abstract

Inappropriate training assessment might have either high social costs and economic impacts, especially in high risks categories, such as Pilots, Air Traffic Controllers, or Surgeons. One of the current limitations of the standard training assessment procedures is the lack of information about the amount of cognitive resources requested by the user for the correct execution of the proposed task. In fact, even if the task is accomplished achieving the maximum performance, by the standard training assessment methods, it would not be possible to gather and evaluate information about cognitive resources available for dealing with unexpected events or emergency conditions. Therefore, a metric based on the brain activity (neurometric) able to provide the Instructor such a kind of information should be very important. As a first step in this direction, the Electroencephalogram (EEG) and the performance of 10 participants were collected along a training period of 3 weeks, while learning the execution of a new task. Specific indexes have been estimated from the behavioral and EEG signal to objectively assess the users' training progress. Furthermore, we proposed a neurometric based on a machine learning algorithm to quantify the user's training level within each session by considering the level of task execution, and both the behavioral and cognitive stabilities between consecutive sessions. The results demonstrated that the proposed methodology and neurometric could quantify and track the users' progresses, and provide the Instructor information for a more objective evaluation and better tailoring of training programs.

Keywords: EEG; brain activity; human factor; human machine interaction; machine learning; training assessment.

Figure 1

NASA Multi Attribute Task Battery (MATB) interface. Emergency lights task (SYSM) in the blue rectangle; Tracking task (TRCK) in the red rectangle; Radio Communication (COMM) in the green rectangle, and Fuel managing task (RMAN) in the orange rectangle (A–F are the fuel tanks).

Figure 2

The images reports an example of triangle by which is possible to quantify the training level of the user. In particular, the area of the triangle, defined by considering the Mean Performance, Performance Stability, and Cognitive Stability as vertices, has been calculated as the sum of the areas of three sub-triangles (Area Triangle #1, Area Triangle #2, and Area Triangle #3). Such triangles were identified by the three indexes and the origin of the coordinate system.

Figure 3

Task performance values over 3 weeks of training. The ANOVA showed a significant (p < 10⁻⁵) improvement of performance from T1 to T5, and then no differences were found between the rest of the sessions. Such results indicated that the subjects reached the task saturation in the session T5. “^**” Means that the statistical significance level (p) is lower than 0.01. Vertical bars denote 0.95 confidence intervals (CI).

Figure 4

Cognitive Stability Index over 3 weeks of training. The ANOVA showed significant differences (p < 10⁻⁴) among the first sessions (T1÷T7), while no differences were found among the last ones (T7÷T12). Such results indicated that the subjects reached the cognitive stability in the session T7. “^**” Means that the statistical significance level (p) is lower than 0.01. Vertical bars denote 0.95 confidence intervals (CI).

Figure 5

Scatter-plots of the correlation between the Mean Performance and the Cognitive Stability. The Pearson's analysis reported significant (p = 0.039) and high correlation (|R| > 0.89) between the measures, as demonstration that their integration could provide the Instructor objective data for the training assessment.

Figure 6

Training metric over 3 weeks of training. The proposed metric takes into account the level of task execution (performance), and both the behavioral and cognitive stabilities. The ANOVA showed significant differences (p < 10⁻⁵) among the first sessions (T1÷T7), while no differences were found among the last ones (T7÷T12). Such results indicated that the subjects reached a training stability in the session T7. “^**” Means that the statistical significance level (p) is lower than 0.01. Vertical bars denote 0.95 confidence intervals (CI).

Figure 7

In order to quantify the users' training level along the training sessions, the behavioral and neurophysiological measures were integrated. Such measures were used to define the sides of a scalene triangle, and then the Area of such a triangle was calculated in each training session as measure of the training level. The Area was normalized with respect to its maximum with the aim to have as references of maximum training the value “1.”

Figure 8

The figure reports the Areas of a representative subject along the considered training sessions. It is possible to appreciate how the Subject 6 improved its behavioral and cognitive skills from the beginning (T1) to the end of the training period (T12).