Medial prefrontal cortex as an action-outcome predictor

Abstract

The medial prefrontal cortex (mPFC) and especially anterior cingulate cortex is central to higher cognitive function and many clinical disorders, yet its basic function remains in dispute. Various competing theories of mPFC have treated effects of errors, conflict, error likelihood, volatility and reward, using findings from neuroimaging and neurophysiology in humans and monkeys. No single theory has been able to reconcile and account for the variety of findings. Here we show that a simple model based on standard learning rules can simulate and unify an unprecedented range of known effects in mPFC. The model reinterprets many known effects and suggests a new view of mPFC, as a region concerned with learning and predicting the likely outcomes of actions, whether good or bad. Cognitive control at the neural level is then seen as a result of evaluating the probable and actual outcomes of one's actions.

Figure 1. (A) The Predicted Response Outcome (PRO) model

In an idealized experiment, a task-related stimulus (S) signaling the onset of a trial is presented. Over the course of a task, the model learns a timed prediction (V) of possible responses and outcomes (r). The temporal difference learning signal (δ) is decomposed into its positive and negative components (ω^P and ω^N, respectively), indicating unpredicted occurrences and unpredicted non-occurrences, respectively. (B) ω^N accounts for typical effects observed in mPFC from human imaging studies. Conflict and error likelihood panels show activity magnitude aligned on trial onset; error and error unexpectedness panels show activity magnitude aligned on feedback. Model activity is in arbitrary units. EL is error likelihood (HEL=High EL; LEL=Low EL). Error bars indicate standard error of the mean (C) Typical time courses for components of the PRO model.

Figure 2. ERP simulations

(A) Left panel: simulated feedback error-related negativity (fERN) difference wave. Effects of surprising outcomes (low error likelihood/error minus high error likelihood/correct) were larger than outcomes which were predictable (high error likelihood/error minus low error likelihood/correct). Right panel: observed ERP difference wave adapted with permission, consistent with simulation results. (B) The effects of speed-accuracy tradeoffs on ERP amplitude are observed in the PRO model (left). Trials for incongruent and congruent conditions were divided into quintile bins by reaction time (large marker indicates slow reaction time, small marker indicates fastreaction time), and activity of the PRO model was calculated for correct trials in each bin. Accuracy and activity of the model were highest for trials with long reaction times, and lowest for trials with short reaction times, consistent with human EEG data (right). (C) The simulated activity of the PRO model (left) reflects amplitude and duration of the N2 component observed in humans EEG studies (right). Adapted with permission.

Figure 3. Single-unit neurophysiology simulation

(A) Calculation of the negative surprise signal ω^N was performed for individual outcome predictions (indexed as i). For predictions of e.g. reward, the surprise signal increases steadily to the time at which the reward is predicted. The signal is suppressed on the occurrence of the predicted reward. Single units predicting error follow a similar pattern, with increased variance in the timing of the error. (B) The complement of negative surprise (i.e. positive surprise ω^P) indicates unpredicted occurrences. (C) Reward-predicting and reward-detecting cells recorded in monkey mPFC consistent with simulation results. The top panel displays activity of a single unit consistent with the prediction of a reward. On error trials, activity peaks and gradually attenuates, potentially signaling an unsatisfied prediction of reward. The bottom panel shows single-unit activity related to the detection of a rewarding event. Adapted with permission.

Figure 4. fMRI simulations

(A) Multiple response effects. The change signal task is modified to require both change and go responses simultaneously when a change signal cue is presented. Change trials lead to greater prediction layer activity (aligned on trial onset) compared with go trials, even though response conflict is by definition absent. The incongruency effect in the absence of conflict is the multiple response effect. (B) Volatility effects. When environmental contingencies change frequently, mPFC shows greater activity. This has been interpreted with a Bayesian model in which mPFC signals the expected volatility, right panel (adapted with permission). In the PRO model, greater volatility in a block led to greater mean ω^N, lower left panel. Surprise signals, in turn, dynamically modulate the effective learning rate of the model (upper left panel), yielding lower effective learning rates during periods of greater stability (F(1,3)=70.3. p=0.00). In the mPFC-lesioned model, learning rates did not significantly change between periods (F(1,3)=0.23, p=0.88). (C) mPFC signals discrepancies between actual and expected outcomes. If errors occur more frequently than correct trials (70% error rate here), mPFC is predicted to show an inversion of the error effect, i.e. greater activity (aligned on feedback) for correct than error trials. (D) Delayed feedback effect. Feedback that is delayed an extra 400 ms on a minority of trials (20% here) leads to timing discrepancies and greater surprise activation (aligned on feedback). (E) Effects of reward salience on error prediction and detection. As rewarding events influence learning to a greater degree, error likelihood effects (aligned on trial onset) decrease while error effects (aligned on feedback) increase. All error bars indicate standard error of the mean.