Temporal characteristics of the influence of punishment on perceptual decision making in the human brain

Abstract

Perceptual decision making is the process by which information from sensory systems is combined and used to influence our behavior. In addition to the sensory input, this process can be affected by other factors, such as reward and punishment for correct and incorrect responses. To investigate the temporal dynamics of how monetary punishment influences perceptual decision making in humans, we collected electroencephalography (EEG) data during a perceptual categorization task whereby the punishment level for incorrect responses was parametrically manipulated across blocks of trials. Behaviorally, we observed improved accuracy for high relative to low punishment levels. Using multivariate linear discriminant analysis of the EEG, we identified multiple punishment-induced discriminating components with spatially distinct scalp topographies. Compared with components related to sensory evidence, components discriminating punishment levels appeared later in the trial, suggesting that punishment affects primarily late postsensory, decision-related processing. Crucially, the amplitude of these punishment components across participants was predictive of the size of the behavioral improvements induced by punishment. Finally, trial-by-trial changes in prestimulus oscillatory activity in the alpha and gamma bands were good predictors of the amplitude of these components. We discuss these findings in the context of increased motivation/attention, resulting from increases in punishment, which in turn yields improved decision-related processing.

Figure 1.

Schematic representation of the behavioral paradigm and behavioral results (n = 22). A, Subjects performed a face-versus-car discrimination task with block-wise punishment manipulation. The punishment condition [0 (blue), 5 (green), or 10 (red)] was indicated at the beginning of each block. Noisy grayscale images of faces and cars, characterized by their percentage phase coherence (low and high), were presented in random order. Each image was presented for 50 ms, followed by an ISI lasting between 2750 and 3250 ms. Subjects were required to make a decision and respond by pressing a button within the first second of the ISI period. B, C, Mean percentage correct performance increased as the amount of punishment increased, whereas mean RTs remained unchanged across the three punishment levels. Error bars indicate ± 1 SEM across participants. D, E, Mean percentage correct performance increased and mean RTs were reduced as the amount of sensory evidence (i.e., image phase coherence) increased.

Figure 2.

Multivariate single-trial analysis reveals temporally specific EEG components related to punishment. Average discriminator performance using a leave-one-out (LOO) cross-validation procedure [Az (LOO)] along the punishment dimension (10-versus-0 punishment levels) for stimulus-locked (A) and response-locked (B) data. The dotted black lines represent the Az leading to a significance level of p = 0.01 (using a bootstrap test). The gray boxes represent time windows with distinct scalp distributions (i.e., forward models, a_τ) as identified using a simple clustering procedure (for details, see Results). Red represents positive correlation between the sensor readings and the extracted discriminating components, whereas blue represents negative correlation. Note, for the stimulus-locked analysis, discrimination performance is significant in the time range 200–470 ms after the stimulus, whereas for response-locked analysis, it was significant in the range 200 ms before to 250 ms after the response.

Figure 3.

Stimulus-locked components associated with punishment. A, Average scalp maps (i.e., average forward models, a_τ) in each of the four stimulus-locked windows as identified in Figure 2A. Red represents positive correlation of the sensor readings to the extracted activity and blue negative correlation. B, Mean discriminator output (y_τ) in each of the four stimulus-locked windows as a function of all three punishment levels [τ indexes the different windows/components, 0 (blue), 5 (green), or 10 (red) punishment levels]. Discriminating projections (w_δ,τ) were estimated using 10-versus-0 punishment levels and subsequently applied to the intermediated (5) punishment level to obtain discriminator output values for all three punishment conditions. Post hoc t tests on y_τ revealed significant differences between 0/5 and 5/10 punishment conditions. C, Significant correlations of the overall accuracy improvements resulting from the presence of punishment (10 + 5 vs 0) and our ability to discriminate between the same conditions using the EEG [i.e., average y_τ (10 + 5) − y_τ(0)] across individual participants, for each of the four stimulus-locked components. D, Temporal profiles for all four stimulus-locked punishment components constructed by applying the discriminating projections (w_δ,τ) estimated for each component (in the window shown by the gray boxes) to an extended time window spanning 200 ms before to 600 ms after the onset of the stimulus averaged across participants. Inset, Temporal profile of the third component from a representative subject.

Figure 4.

Trial-by-trial correlation between behavioral choices and discriminator output. A, Beta values from multiple logistic regression in which trial-by-trial fluctuations around the mean discriminator output (computed for each punishment level separately) from our four punishment components were used to predict participants' trial-by-trial choices (i.e., probability correct). Single-trial changes in discriminator output (y_τ) were only significantly (*) predictive of choice for the last two stimulus-locked components. Error bars indicate ± 1 SEM. B, Probability correct versus discriminator output (y_τ) on binned data but with fits resulting from four separate (one for each component) single-trial regression models using the beta values estimated in A.

Figure 5.

Response-locked components associated with punishment (same conventions as in Fig. 3). A, Average scalp maps in each of the three response-locked components as identified in Figure 2B. Scalp maps represent forward models of the discriminating activity across participants. Red represents positive correlation of the sensor readings to the extracted activity and blue negative correlation. B, Mean discriminator output (y_τ) in each of the three components as a function of all punishment levels. C, Correlations of the overall accuracy improvements resulting from the presence of punishment (10 + 5 vs 0) and our ability to discriminate between the same conditions using the EEG [i.e., average y_τ (10 + 5) − y_τ(0)] across individual participants, for each of the three response-locked components. D, Temporal profiles for all three response-locked punishment components in a time window spanning 400 ms before to 300 ms after the response averaged across participants.

Figure 6.

Multivariate single-trial analysis reveals temporally specific EEG components related to the amount of sensory evidence. Average discriminator performance using a leave-one-out (LOO) cross-validation procedure [Az (LOO)] along the sensory evidence dimension (high-versus-low image phase coherence) for stimulus-locked (A) and response-locked (B) data. The dotted black lines represent the Az leading to a significance level of p = 0.01 (using a bootstrap test). The gray boxes represent time windows with distinct scalp distributions (i.e., forward models, a_τ). Red represents positive correlation between the sensor readings and the extracted discriminating components, whereas blue represents negative correlation. Note, for the stimulus-locked analysis, discrimination performance is significant in the time range 160–200 and 340–475 ms after the stimulus, whereas for response-locked analysis, it is significant in the range 110–80 ms before the response.

Figure 7.

Stimulus- and response-locked components associated with sensory evidence. A, Average scalp maps in each of the two stimulus-locked components (early and late) as identified in Figure 6A (top row). Temporal profiles for the two stimulus-locked components (bottom row, gray: low sensory evidence; black: high sensory evidence). Time traces were constructed by applying the discriminating projections (w_δ,τ) estimated for each component (in the window shown by the gray boxes) to an extended time window spanning 250 ms before to 550 ms after the onset of the stimulus. B, Average scalp map for the response-locked component as identified in Figure 6B (top). Temporal profile for the response-locked component in a time window spanning 400 ms before to 300 ms after the response (bottom) averaged across participants. Scalp maps in A and B represent forward models of the discriminating activity across participants. Red represents positive correlation of the sensor readings to the extracted activity and blue negative correlation.

Figure 8.

Overlap of late sensory evidence and punishment effects. The discriminator output of the late sensory evidence component was used to stratify trials based on punishment. On average, late sensory evidence activity appeared to be modulated by punishment. Furthermore, the mean discriminator amplitude for the high punishment condition (10) was significantly higher than that of the no punishment condition (p < 0.05). Error bars indicate ± 1 SEM across participants.

Figure 9.

Prestimulus oscillatory activity associated with punishment. Spectral amplitudes in baseline alpha (A) and gamma (B) frequency bands are parametrically modulated by punishment. Scalp maps indicate the slope of change in spectral amplitudes as a function of the amount of punishment (using linear regression), estimated for each sensor separately. Black dots indicate sensors for which the slope is significantly different from 0 (top row). For illustration purposes, the corresponding mean spectral amplitudes over significant sensors are plotted as a function of punishment (bottom row). Negative slopes indicate a reduction in spectral amplitude as punishment increases (frontal and occipitoparietal sensors for alpha, frontal sensors for gamma), whereas positive slopes indicate an increase in spectral amplitude with punishment (occipitotemporal sensors for gamma).