A common mechanism for adaptive scaling of reward and novelty

Abstract

Declarative memory is remarkably adaptive in the way it maintains sensitivity to relative novelty in both unknown and highly familiar environments. However, the neural mechanisms underlying this contextual adaptation are poorly understood. On the basis of emerging links between novelty processing and reinforcement learning mechanisms, we hypothesized that responses to novelty will be adaptively scaled according to expected contextual probabilities of new and familiar events, in the same way that responses to prediction errors for rewards are scaled according to their expected range. Using functional magnetic resonance imaging in humans, we show that the influence of novelty and reward on memory formation in an incidental memory task is adaptively scaled and furthermore that the BOLD signal in orbital prefrontal and medial temporal cortices exhibits concomitant scaled adaptive coding. These findings demonstrate a new mechanism for adjusting gain and sensitivity in declarative memory in accordance with contextual probabilities and expectancies of future events.

Figure 1

Experimental designs. One of three possible cues (colored squares) indicated either which of two possible reward values (Experiments I and II) (A, C) or which of two possible degrees of novelty associated with a scene picture (Experiment III) (D, E) could follow with equal probability. Following the reward (Experiments I and II) (A–C) or scene picture (Experiment III) (D, E) subjects indicated whether they saw the higher or the lower of the two possible reward values (Experiments I and II), or the contextually more novel or familiar of two possible images (Experiment III). Additionally, in Experiment II, the reward value was followed by a novel scene picture, which had to be correctly classified as indoor or outdoor to receive the reward (C). fMRI was acquired during Experiments I and III. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 2

Parametric modulation. In both fMRI experiments, hemodynamic responses at outcome of reward (left, Experiment I) and novelty (right, Experiment III) were analyzed using parametric modulation. The parametric modulators tested expressed “scaled adaptive coding,” “absolute coding,” and “linear prediction error.” The parametric modulator of “adaptive scaling” reflects a binary response (in other words a scaled prediction‐error signal) to the higher versus lower of the two possible rewards, or more novel versus more familiar of images; the parametric modulator for “absolute value” reflects either the absolute value of the reward outcome or the absolute novelty status of a picture (as defined by the reciprocal of the number of repetitions); the parametric modulator of a “linear prediction error” reflects the unscaled prediction error responses to reward/novelty outcomes based on the difference between the received reward and the mean expected reward as signaled by the cue (Experiment I), or the difference between the novelty status of the presented image and the mean expected novelty status as signaled by the cue (Experiment III; see Materials and Methods). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 3

fMRI results (Experiment I). Hemodynamic responses for reward outcome expressed scaled adaptive coding in medial temporal lobes including hippocampus and rhinal cortex (A), bilateral ventral striatum (B), and the medial and orbital prefrontal cortex (C). Note that parameter estimates reflect the fit of the different parametric modulators (“scaled adaptive coding,” “absolute coding,” “linear PE”); within the depicted regions only the regressor for “scaled adaptive coding” significantly explained variance of the hemodynamic responses (greater than zero, P < 0.05; error bars denote one standard error of the mean). Furthermore, direct comparison shows significant differences between parametric modulators expressing “scaled adaptive coding” and “linear PE” or “absolute coding” (P < 0.05, two‐tailed). Activation maps were superimposed on a T1‐weighted MNI standard brain, coordinates are given in MNI space, and color bar indicates T‐values. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 4

Recognition memory performance 1 day after encoding (Experiment II). The corrected hit‐rate (corrected remember estimates + corrected familiarity estimates) for novel pictures followed the pattern of adaptive scaling. Medium reward value (£0.50) improved learning when it was the higher possible reward compared to when it was the lower possible reward (indicated by the asterisk, P < 0.05, two‐tailed). Error bars denote one standard error of the mean.

Figure 5

fMRI results (Experiment III). Hemodynamic responses for novelty outcome expressed scaled adaptive coding within the hippocampus (A), rhinal cortex (B), and orbital parts of the mPFC (C). Parameter estimates reflect the fit of the different parametric modulators (“scaled adaptive coding,” “absolute coding,” “linear PE”); within the depicted regions, only the regressor for “scaled adaptive coding” significantly explained variance of the hemodynamic responses (indicated by asterisk). Direct comparison shows significant differences between parametric modulators expressing “scaled adaptive coding” and “linear PE” or “absolute coding” (P < 0.05, two‐tailed; except “adaptive scaling” vs. “linear PE” in rhinal cortex, P < 0.05, one‐tailed). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 6

Common areas expressing scaled adaptive coding of reward and novelty. As identified by between‐subject ANOVA and implicit masking, overlapping regions within the hippocampus (A), rhinal cortex (B), and an orbital part of the mPFC (C) expressed scaled adaptive coding of reward as well as novelty. Note that parameter estimates reflect the fit of the parametric modulator (“scaled adaptive coding”). As indicated by the asterisk within the depicted regions, only the parametric modulator expressing “scaled adaptive coding” significantly explained variance of the hemodynamic responses. Error bars denote one standard error of the mean. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]