Can EEG Be Adopted as a Neuroscience Reference for Assessing Software Programmers' Cognitive Load?

Abstract

An emergent research area in software engineering and software reliability is the use of wearable biosensors to monitor the cognitive state of software developers during software development tasks. The goal is to gather physiologic manifestations that can be linked to error-prone scenarios related to programmers' cognitive states. In this paper we investigate whether electroencephalography (EEG) can be applied to accurately identify programmers' cognitive load associated with the comprehension of code with different complexity levels. Therefore, a controlled experiment involving 26 programmers was carried. We found that features related to Theta, Alpha, and Beta brain waves have the highest discriminative power, allowing the identification of code lines and demanding higher mental effort. The EEG results reveal evidence of mental effort saturation as code complexity increases. Conversely, the classic software complexity metrics do not accurately represent the mental effort involved in code comprehension. Finally, EEG is proposed as a reference, in particular, the combination of EEG with eye tracking information allows for an accurate identification of code lines that correspond to peaks of cognitive load, providing a reference to help in the future evaluation of the space and time accuracy of programmers' cognitive state monitored using wearable devices compatible with software development activities.

Keywords: bio-signal processing; biofeedback; electroencephalogram; human error; software engineering.

Figure 1

Representative schematics of one trial procedure, involving the fixed cross, in a screen, before and after the relevant tasks for analysis, i.e., the reading control task and the code comprehension task.

Figure 2

Complexity level of each code tasks according to each one of the five software complexity metrics used.

Figure 3

Schematic representation of the acquisition setup.

Figure 4

Examples of some artifact components that were identified and removed: (i) In component IC1, an eye blinking artifact component can easily be recognized; (ii) component IC2 contains cardiac artifact; (iii) component IC3 shows another example of ocular artifacts, the saccades/microsaccades; and (iv) component IC10 contains involuntary muscle movement.

Figure 5

Schematic representation of the feature normalization and transformation steps.

Figure 6

p-values of the pairwise comparison of the classification model options, with Bonferroni correction. The values higher than the significance level of 0.05 are colored with light blue, while the intermediate blue is related to a significance level of 0.05 and dark blue to a significance level of 0.01. PCA: Principal Component Analysis; Krusk: Kruskal-Wallis H test; NMI: Normalized Mutual Information; Reli: ReliefF Algorithm; SVM: Support Vector Machine; FLDA: Fisher Linear Discriminant Analysis classifier; kNN: k-nearest neighbors algorithm; Naive: Naive Bayes classifier.

Figure 7

Mental effort felt by the participants and written on the NASA-TLX for the three different Code tasks.

Figure 8

Topographic map representing the percentage of the features corresponding to each electrode after feature selection with Kruskal–Wallis H test, for the multiclass scenario C1 vs. C2 vs. C3 vs. Control.

Figure 9

Radar plot depicting which type of features are more frequent (in %) in the dataset obtained after feature selection, for the multiclass scenario C1 vs. C2 vs. C3 vs. Control.

Figure 10

Example of the fusion of EEG with eye tracking, for an intermediate participant during Code task 1. (A) Density of gaze points with the red line as a reference of 50% of the maximum row density; (B) clusters of gaze points over time and the y-axis; (C) code task figure overlapped with gaze points; and (D) discriminant EEG features values over time.

Figure 11

Example of the fusion of EEG with eye tracking, for an expert participant during the Code task 2. (A) Density of gaze points, with the red line as a reference of 50% of the maximum row density; (B) clusters of gaze points over time and the y-axis; (C) code task figure overlapped with gaze points; and (D) discriminant EEG features values over time.