Neurophysiological encoding of aversive prediction errors

Abstract

Aversive prediction error (PE) brain signals generated by unexpected pain or pain absence are crucial for learning to avoid future pain. Yet, the detailed neurophysiological origins of PE signaling remain unclear. In this study, we combined an instrumental pain avoidance task with computational modeling and magnetoencephalography to detect time-resolved activations underlying pain expectations and aversive PE signals in the human brain. The task entailed learning probabilistically changing cue-pain associations to avoid receiving a pain stimulus. We used an axiomatic approach to identify general aversive PE signals that encode the degree to which the outcome deviated from expectations. Our findings indicate that aversive PE signals are generated in the alpha band (8-12 Hz) by the midbrain/diencephalon, lateral orbitofrontal cortex, and ventrolateral prefrontal cortex approximately 150 milliseconds after outcome delivery. Moreover, alpha oscillations in these regions also encoded pain expectations before the outcome. We speculate that this may facilitate the rapid generation of PEs by allowing outcome-related nociceptive activity to be integrated with ongoing predictive signals. Finally, decisions to avoid pain recruited alpha oscillations in the anterior cingulate and dorsomedial prefrontal cortices, suggesting their active engagement in comparing predicted action values. Overall, our data reveal the rapid neurophysiological mechanisms underlying the generation of aversive PEs and subsequent decision-making.

Trial registration: ClinicalTrials.gov NCT04603417.

Keywords: Aversive prediction error; Brain oscillations; Magnetoencephalography; Pain-related learning.

Figure 1.

Instrumental pain avoidance task and behavioral results. (A) One experimental trial. Participants had 1.8 seconds to choose between 2 visual objects (using mouse clicks) associated with different probabilities of receiving a pain or no-stimulus outcome. Their choice was confirmed with a display of the selected object that lasted another 0.2 seconds. After an expectation (anticipatory) period of 2 seconds marked by a “+” plus symbol displayed on the screen, participants either received a painful stimulus or no-stimulus. The outcome (pain/no-stimuli) period was marked by a “*” asterisk fixation-point on the screen. If pain outcome was received, participants rated their pain using a visual analog scale. Trials were separated by a 1- to 3-second intertrial interval (ITI). (B) Random walk and choice selection from one participant. The blue and orange lines depict the probability of pain associated with each visual item over the 150 trials (one of 2 possible pairs of random walks used in the study). At the bottom of the curves, blue and orange dots represent the choices made by the participant, and black triangles represent pain outcomes. (C) Hypothesized aversive PE signal. PEs represent the difference between outcome and expectation. Left panel: Expectations preceding outcome delivery. Middle panel: Pain outcome is more aversive than no-stimulus outcome. Right panel: PEs linearly encode the relative aversiveness of the outcome, from the least aversive (no-stimulus when pain was expected) to the most aversive (pain when no-stimulus was expected). Therefore, axioms for an aversive prediction include (1) higher activity for pain vs no-stimulus outcome, regardless of expectations (axiom#1), and (2) higher activity for low vs high pain expectancy, regardless of outcome (axioms#2a-expectancy effects within pain trials and 2b-expectancy effects within no–stimulus trials). (D) Logistic regression model results. Probability of switching to the other possible choice as a function of pain received 1 to 6 trials back shown as mean odds ratio of both trial sets. Error bars represent standard error of the mean. Odds ratio > 1 indicates high likelihood of switching.

Figure 2.

Model-derived aversive prediction error (PE) correlates of evoked responses. (A) Evoked responses when pain and no-stimulus outcome were delivered to right hand (RH). Top and bottom panel, respectively, show the butterfly plot of grand averaged evoked responses from MEG gradiometer channels from pain and no-stimulus delivery. The data sensor topographies derived from the peaks at specific latencies are shown above each panel. Strength of magnetic fields is represented in femto-Tesla (fT/cm). Black vertical line denotes the outcome onset cue that is, asterisk presentation on the screen. Red vertical line denotes the onset of pain outcome. Red horizontal line plot in each panel shows event-related potentials (microvolts) from an EEG channel (Cz − Avg for pain outcome and Pz − Avg for no-stimulus outcome) depicting the positive and negative deflections. It is worth noting that no-stimulus (absence of pain) also resulted in low amplitude responses. (B) Cortical correlates of model-derived aversive PEs. Evoked activity conjunction of LH and RH trial sets (conjunction of supplementary Figs. S3 and S4, http://links.lww.com/PAIN/C325) correlated with reinforcement learning model–derived PE estimates at outcome delivery cluster-thresholded (P < 0.05, corrected for family-wise error rate (FWER), two-tailed) with cluster-defining thresholds of P < 0.001. The color mapping indicates the cumulative duration during which the sources were correlated between 36 and 235 milliseconds after outcome delivery (shaded region in Fig. 2A top and bottom panel). For example, yellow regions correlated for greater duration than red regions. The inset line plot shows activity profile derived from an exemplar region indicating the negative association between evoked amplitude and PEs. For pain and no-stimulus outcomes, mean responses from evoked source time series (arbitrary units) were extracted between 36 and 235 milliseconds after outcome delivery by quartiles of model-derived PE estimates. RH and LH trial sets were further averaged. Error bars represent SEM.

Figure 3.

Model-derived aversive prediction error (PE) correlates of oscillatory responses. (A) TF responses from average source time series from the contralateral primary somatosensory cortex (S1) when pain and no-stimulus outcome were delivered to RH. Black vertical line denotes the outcome onset cue that is, asterisk presentation on the screen. Red vertical line denotes the onset of pain outcome. Frequency bands noted are alpha [8 12] Hz, beta [13 30] Hz, low gamma [30 60] Hz, and high gamma [61 90] Hz. TF maps were subjected to decibel (dB) normalization over the baseline period (1 second before the start of each trial ie, presentation of 2 visual objects in Fig. 1A). The line plot shows activity profile derived from an exemplar region indicating the negative association between alpha power and PEs. For pain and no-stimulus outcomes, mean responses from relative alpha oscillatory power (db) were extracted from 150 to 300 milliseconds after outcome delivery by quartiles of model-derived PE estimates. RH and LH trial sets were further averaged. Error bars represent SEM. (B) Cortical correlates of model-derived aversive PEs in different frequency bands. Oscillatory activity conjunction of LH and RH trial sets (conjunction of supplementary Figs. S5 and S6, http://links.lww.com/PAIN/C325) correlated with reinforcement learning model–derived PE estimates at outcome delivery cluster-thresholded (P < 0.05, corrected for family-wise error rate (FWER), two-tailed) with cluster-defining thresholds of P < 0.01. Sources are color coded according to the cumulative duration they were correlated between 0 and 1 second after pain onset (eg, yellow regions correlated for greater duration than red regions).

Figure 4.

Aversive PE correlates satisfying axiomatic tests. (A) Spatial correlates of aversive PE: Oscillatory activity conjunction of LH and RH sets (conjunction of supplementary Figs. S9 and S10, http://links.lww.com/PAIN/C325) for alpha band cluster-thresholded (P < 0.05, FDR corrected, one-tailed) with cluster-defining thresholds of P < 0.01. Conjunction of LH and RH sets did not yield significant clusters in the beta or gamma bands. Three main regions are labeled. MB/DC, midbrain/diencephalon; lOFC, lateral orbitofrontal cortex; vlPFC, ventrolateral prefrontal cortex; R, right hemisphere. (B) Temporal mapping of aversive PE in each of the identified regions of interest (ROIs). Time 0 represents the onset of pain. Note: the average FWHM of the wavelet used is ∼300 milliseconds in the alpha band; hence, any activity 150 milliseconds after pain onset represent true poststimulus activity arising from outcome delivery. The frequency resolution in the alpha band is ∼1.5 Hz. (C) Activity profile derived from 2 select ROIs (top and middle panel) that satisfied the axioms (pain [red] > no-stimulus [blue] and parametric variation of oscillatory activity with model-derived expectancy—on pain trials and no-stimulus trials). For pain and no-stimulus outcomes, mean responses were extracted by quartiles of model-derived pain expectancy estimates. RH and LH trial sets were further averaged. Error bars represent SEM. Effects of expectancy on pain ratings (bottom panel) from pain-reinforced trials. Average of LH and RH trial sets. Median with first and third quartiles depicted as error bars. The significance of the slope (P = 0.0060) was tested using a nonparametric permutation test after generating a surrogate distribution by shuffling the order of pain rating w.r.t expectancies in each participant 1000 times.

Figure 5.

Expectancy effects on outcome anticipation. (A) Spatial correlates of expectancy effects during outcome anticipation. Oscillatory activity conjunction of LH and RH sets (conjunction of supplementary Figs. S12 and S13, http://links.lww.com/PAIN/C325) cluster-thresholded (P < 0.05, FDR corrected, two-tailed) with cluster-defining thresholds of P < 0.01. Color bar indicates the cumulative duration in secs over which the sources were correlated within the 2-second anticipatory period. Only selected regions of interest (ROIs) that exhibited prolonged correlation within each frequency band are labeled. For the gamma band, the right hemisphere correlates are from low gamma, whereas the left hemisphere correlates are from high gamma band. R indicates right and L indicates left hemispheres. Refer to text for ROI abbreviations. (B) Temporal mapping of expectancy effects in labelled ROIs in each frequency band. Time 0 represents the onset of anticipatory period that lasted for 2 seconds. The anticipatory period was terminated at outcome onset. (C) Time-frequency plot from right lateral orbitofrontal (R lOFC) during anticipatory period. Dotted vertical black line is the onset of expectation (display of + sign on the screen after choice decision), and dotted red vertical line is the termination of expectation at the onset of outcome. TF maps were subjected to decibel (dB) normalization over the baseline period (1 second before the start of each trial ie, appearance of 2 visual objects in Fig. 1A). Note: the average FWHM of the wavelet used in the alpha, beta, low gamma, and high gamma bands were ∼300 milliseconds, ∼150 milliseconds, ∼60 milliseconds, and <50 milliseconds, respectively. Hence, any activity before 150 milliseconds (alpha), 75 milliseconds (beta), 30 milliseconds (low gamma), and 25 milliseconds (high gamma) before outcome onset cue (2 seconds) represent true activity arising from expectations. The average frequency resolution in the alpha, beta, low gamma, and high gamma bands are ∼1.5 Hz, ∼3 Hz, ∼6.5 Hz, and >8.5 Hz, respectively. (D) Profiles of activity derived R lOFC and right midbrain/diencephalon (R MB/DC) in the alpha band (average of LH and RH trial sets) show activity correlating negatively with model-derived expectancy estimates (greater oscillatory power for low expected pain). Mean responses were extracted by quartiles of model-derived pain expectancy estimates. Error bars represent SEM. Numbers in red indicate the percentage of trials preceded by pain outcome in the previous trial, showing low pain expectations were preceded predominantly by no-stimulus outcome, and high pain expectations were preceded predominantly by pain outcome.

Figure 6.

Transition from expectations to prediction errors (PEs). Time resolved activity in the alpha band encoding pain expectancy and aversive PE in right lateral orbitofrontal (R lOFC) and right midbrain/diencephalon (R MB/DC). These regions encoded both pain expectation and aversive PEs in the alpha band. Horizontal bar plots (bottom row) show the time interval during which the regions conformed to the axiomatic tests indicated in Fig. 1C. Cortical maps (middle row) show the sources generating the activity at the specified time periods. Activity profiles (top row) show the alpha event-related power changes that conformed to axiomatic tests. *when axioms were satisfied with significance.

Figure 7.

Correlates of choice-difficulty and decision-making. Oscillatory activity conjunction of LH and RH trial sets (conjunction of supplementary Figs. S14 and S15, http://links.lww.com/PAIN/C325) cluster-thresholded (P < 0.05, FDR corrected, two-tailed) with cluster-defining thresholds of P < 0.01 in the alpha and high gamma bands. Time periods next to frequency bands are relative to the appearance of 2 visual objects on the screen. The line plots show the profiles of activity derived from the labeled regions of interest (LH and RH sets averaged) modulated by choice difficulty after pain (red) and no-stimulus outcome (blue). Mean responses were extracted by quartiles of model-derived choice difficulty estimates. Error bars represent SEM. Red and blue numbers on the top, respectively, indicate the percentage of trials in which participants decided to switch choices after receiving pain and no-stimulus outcomes. As expected, choice difficulty was mostly driven by pain outcomes. No-stimulus outcomes resulted in easy choice decision to stay (>99% of trials), whereas pain outcomes resulted in difficult decision to either stay or switch. Greater choice difficulty after pain outcome led to greater proportions of switch decisions, compared to easier choice decisions.