On the relative importance of attention and response selection processes for multi-component behavior - Evidence from EEG-based deep learning

Abstract

Goal-directed behavior often requires concatenating different actions to achieve a goal. The neural correlates of such multi-component behavior have extensively been investigated. However, it is still enigmatic whether it is possible to predict, using single-trial EEG data and on a single-subject level, that an individual is confronted with a situation imposing high or low demands on multi-component behavior. This study gathered data from N = 239 individuals and applied EEG-based deep learning combined with explainable artificial intelligence, temporal EEG signal decomposition, and source localization. We show that attentional selection and sensory integration processes in sensory association cortices were highly predictive with ∼86%. Processes specifying rule-based response selection and translation, associated with superior and posterior parietal cortices, were also predictive with ∼70%. This, however, was only possible when the information about sensory integration was not available for deep learning. Therefore, sensory integration processes are particularly important in the decoding of whether an individual is confronted with a situation imposing high or low demands on response selection capacity limited multi-component behavior. The results provide insights into the relative importance of various cognitive processes during complex goal-directed behavior and suggest that attentional processes are important to consider during multi-component behavior.

Keywords: Attention; Deep learning; EEG; Response selection.

Fig. 1

Illustration of the stop-change paradigm. GO trials end with the first response given. Stop-change (SC) trials require stopping of the GO response as soon as the STOP signal is presented. The CHANGE signal is presented with a stop-change delay (SCD) i.e. stimulus onset asynchrony (SOA) of either 0 ms or 300 ms relative to the STOP signal. The CHANGE indicates a change in reference line. SC trials end after any response to the CHANGE stimulus.

Fig. 2

Undecomposed data classification accuracy and confusion matrix. A) The classification accuracy for the two-fold problem for each individual subject is shown. Individual subjects are shown on the x-axis, sorted by accuracy. The y-axis denotes the classification accuracy. The horizontal black line denotes the chance level. B) Classification accuracy per fold is shown (x-axis). The y-axis denotes the classification accuracy. The plus denotes the specific chance levels as determined using the method by Combrisson and Jerbi (2015b). The dark grey bar shows the mean classification accuracy of folds 1 to 10, the error bars denote the upper and lower bounds of the 95 confidence interval the upper lower confidence bounds. C) Confusion matrix showing the classification results for SCD0 and SCD300 conditions. The colors and numbers in the matrix denote the frequency at which the real data was classified (y-axis) into one of the two possible predicted classes (x-axis). . (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

Fig. 3

Results of the deep learning analysis for undecomposed data. A) Saliency maps depicting the relevance of each datapoint and electrode for classification between the two classes. Higher values indicate that the specific feature at the respective time point contributed more to classification accuracy. The x-axis denotes the time in seconds relative to STOP signal onset. B) Saliency values for electrode C6 as well as a pool of parietal electrodes (PO3, PO4, PO7, PO8, P7, and P8) are shown (primary left x-axis). Cohens'd effect sizes are given on the secondary (right) y-axis, depicting the effect sizes of the comparison of C6 and the pooled parietal electrodes by means of a dependent sample t-test. The x-axis denotes the time in seconds relative to STOP signal onset. C) Event-related potential plots of electrode C6 of SCD0 and SCD300 condition. The x-axis denotes the time in seconds relative to STOP signal onset. The grey shaded areas depict the peaks of the N1 in the SCD0 and SCD300 condition. D) Top: Scalp topography plots denoting amplitude distribution in the respective N1 time window of SCD0 and SCD300 condition. Below: Results of the sLORETA source localization for SCD0 and SCD300 condition. Only significant activations corrected for multiple comparisons are shown (p < .05).

Fig. 4

RIDE S-cluster classification accuracy and confusion matrix. A) The classification accuracy for the two-fold problem for each individual subject is shown. Individual subjects are shown on the x-axis, sorted by accuracy. The y-axis denotes the classification accuracy. The horizontal black line denotes the chance level. B) Classification accuracy per fold is shown (x-axis). The y-axis denotes the classification accuracy. The plus denotes the specific chance levels as determined using the method by Combrisson and Jerbi (2015b). The dark grey bar shows the mean classification accuracy of folds 1 to 10, the error bars denote the upper and lower bounds of the 95 confidence interval. C) Confusion matrix showing the classification results for SCD0 and SCD300 conditions. The colors and numbers in the matrix denote the frequency at which the real data was classified (y-axis) into one of the two possible predicted classes (x-axis). . (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

Fig. 5

Results of the deep learning analysis for the RIDE S-cluster data. A) Saliency maps depicting the relevance of each datapoint and electrode for classification between the two classes are shown. Higher values indicate that the specific feature at the respective time point contributed more to classification accuracy. The x-axis denotes the time in seconds relative to the STOP signal onset. B) Saliency values for electrode C6 as well as a pool of parietal electrodes (PO3, PO4, PO7, PO8, P7, and P8) are shown (primary left x-axis). Cohens'd effect sizes are given on the secondary (right) y-axis, depicting the effect sizes of the comparison of C6 and the pooled parietal electrodes by means of a dependent sample t-test. The x-axis denotes the time in seconds relative to the STOP signal onset. C) Event-related potential plots of electrode C6 of SCD0 and SCD300 condition are shown. The x-axis denotes the time in seconds relative to the STOP signal onset. The grey shaded areas depict the peaks of the S-cluster N1 in the SCD0 and SCD300 condition. D) Top: Scalp topography plots denoting amplitude distribution in the respective S-cluster N1 time window of SCD0 and SCD300 condition. Below: Results of the sLORETA source localization for SCD0 and SCD300 condition. Only significant activations corrected for multiple comparisons are shown (p < .05).

Fig. 6

RIDE C-cluster classification accuracy and confusion matrix. A) The classification accuracy for the two-fold problem for each individual subject is shown. Individual subjects are shown on the x-axis, sorted by accuracy. The y-axis denotes the classification accuracy. The horizontal black line denotes the chance level. B) Classification accuracy per fold is shown (x-axis). The y-axis denotes the classification accuracy. The plus denotes the specific chance levels as determined using the method by Combrisson and Jerbi (2015b). The dark grey bar shows the mean classification accuracy of folds 1 to 10, the error bars denote the upper and lower bounds of the 95 confidence interval. C) Confusion matrix showing the classification results for SCD0 and SCD300 conditions. The colors and numbers in the matrix denote the frequency at which the real data was classified (y-axis) into one of the two possible predicted classes (x-axis). . (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

Fig. 7

Results of deep learning analysis for RIDE C-cluster data. A) Saliency maps depicting the relevance of each data point and electrode for classification between the two classes. Higher values indicate that the specific feature at the respective time point contributed more to classification accuracy. The x-axis denotes the time in seconds relative to the STOP signal onset. B) Event-related potential plot at electrode PO3 of SCD0 and SCD300 condition. The x-axis denotes the time in seconds relative to the STOP signal onset. The grey shaded areas depict the time window of highest saliency for SCD0 and SCD300 condition. C) Top: Scalp topography plots denoting amplitude distribution in the time window of highest saliency of SCD0 and SCD300 condition. Below: Results of the sLORETA source localization for SCD0 and SCD300 condition in the time window of highest saliency. Only significant activations corrected for multiple comparisons are shown (p < .05).