Value-Driven Adaptations of Mesolimbic Dopamine Release Are Governed by Both Model-Based and Model-Free Mechanisms

Abstract

The magnitude of dopamine signals elicited by rewarding events and their predictors is updated when reward value changes. It is actively debated how readily these dopamine signals adapt and whether adaptation aligns with model-free or model-based reinforcement-learning principles. To investigate this, we trained male rats in a pavlovian-conditioning paradigm and measured dopamine release in the nucleus accumbens core in response to food reward (unconditioned stimulus) and reward-predictive conditioned stimuli (CS), both before and after reward devaluation, induced via either sensory-specific or nonspecific satiety. We demonstrate that (1) such devaluation reduces CS-induced dopamine release rapidly, without additional pairing of CS with devalued reward and irrespective of whether the devaluation was sensory-specific or nonspecific. In contrast, (2) reward devaluation did not decrease food reward-induced dopamine release. Surprisingly, (3) postdevaluation reconditioning, by additional pairing of CS with devalued reward, rapidly reinstated CS-induced dopamine signals to predevaluation levels. Taken together, we identify distinct, divergent adaptations in dopamine-signal magnitude when reward value is decreased: CS dopamine diminishes but reinstates fast, whereas reward dopamine is resistant to change. This implies that, respective to abovementioned findings, (1) CS dopamine may be governed by a model-based mechanism and (2) reward dopamine by a model-free one, where (3) the latter may contribute to swift reinstatement of the former. However, changes in CS dopamine were not selective for sensory specificity of reward devaluation, which is inconsistent with model-based processes. Thus, mesolimbic dopamine signaling incorporates both model-free and model-based mechanisms and is not exclusively governed by either.

Keywords: behavior; dopamine; learning; motivation; nucleus accumbens; rats.

Figure 1.

Behavior during devaluation test and pavlovian conditioning. A, Left, Schematic of operant task in which a lever is extended for FI5, requiring a press after this period for the delivery of a food pellet in the food magazine. The task concluded after the completion of 15 trials or after 15 min, whichever occurred first. Right, Motivation to lever press for reward was tested before (blue) and after (red) a 30 min feeding period. On separate days, the rats received either no food (mock) or 20 g of pellets or chow within the operant chamber (exposure to each feeding regimen counterbalanced across animals). Chow is the homecage diet, whereas pellets are only presented in the operant chamber. B, Caloric intake during pellet- or chow-feeding periods. C, Lever presses during the FI5 period in hungry (blue) and sated (red) states across mock, pellet, and chow-feeding conditions. D, In our pavlovian task, a 5 s audio-tone cue (3 and 5 kHz tones; counterbalanced) is presented on a vITI30 schedule, followed by either a single pellet (CS+) or no reward (CS−). E, Food-magazine head–entry rate during the ITI (black), CS+ (blue), and CS− (green) presentations across conditioning sessions. F, The percentage of trials with a food-magazine head–entry during the ITI (5 s prior to cue onset) and cue epochs. G, Latency of food-magazine head–entry upon cue onset. *p < 0.05; **p < 0.01.

Figure 2.

Dopamine response to CS+ is rapidly reduced after satiety. A, Trial-based schematic where consecutive CS+ presentations (without reward pairing) are evaluated pre- (green, blue) and post- (red) feeding. B, Caloric intake during pellet- or chow-feeding periods. C, Histological verification of electrode position in the nucleus accumbens core (n = 24 electrodes, 20 animals). D, Top, Representative color plot and dopamine response to CS+. Dotted lines denote the CS+ onset and offset in an omission trial where no reward was given. Bottom left, Current versus time trace shows dopamine release in response to the CS+ (yellow indicates duration of CS+ presentation). Bottom right, Cyclic voltammogram confirms the detection of dopamine. E, Electrode sensitivity to dopamine, prompted by unexpected single-pellet deliveries before the start of recording sessions, remains stable across sessions. F, Dopamine release for CS+ presentations in mock (left), pellets (middle), and chow (right) feeding conditions. Dopamine concentrations for pre- and postfeeding CS+ presentations are shown as insets. G, Dopamine concentration compared across feeding conditions for pre-CS+ 1, pre-CS+ 2, and post-CS+ presentations. While consistent across treatments before devaluation, dopamine response to CS+ after devaluation significantly decreased after pellet and chow consumption compared with mock treatment. H, Dopamine concentration per trial for pre-CS+ 1, pre-CS+ 2, and post-CS+ presentations across feeding treatments. *p < 0.05; **p < 0.01; ***p < 0.0001; ^#p < 0.1; ^$p < 0.05 (chow, T1 vs T2); ns, not significant.

Figure 3.

CS+ dopamine is rapidly restored by reintroduction of CS+/US pairings. A, Trial-based schematic where pre- (blue) and post- (red) conditioning phases were evaluated. Post-CS+ trials (dotted line) were used to compare results presented in Figure 2. The conditioning phases consisted of 50 trials, in which 70% were CS+ and US paired trials (35 trials), 10% CS+ (no pellet; 5 trials), 10% US (unpredicted; 5 trials), and 10% CS− (5 trials). Trials were presented in pseudorandomized order on a vITI30 schedule. B, Caloric intake during pellet- or chow-feeding periods. C, Dopamine release across CS trials (5 trials) during conditioning in mock (left), pellet (middle), and chow (right) feeding conditions. Dopamine peak concentrations are shown as insets. Yellow box denotes CS duration. D, Dopamine concentration in CS+ trials during preconditioning (left) and postconditioning (middle). By the first CS+-only trial (right), dopamine has already increased relative to post-CS+ pellet (Fig. 2F, middle, dotted line), indicating that CS+ response rapidly restores after pellet and chow-feeding treatment. E, Dopamine concentration to unpredicted US presentation (5 trials) during conditioning in mock (left), pellet (middle), and chow (right) feeding conditions. Dopamine is shown in insets. F, Dopamine in response to the first pellet exposure (either CS+/US pairing or US-only trial) and subsequent US-only trials during postconditioning across feeding conditions. G, Left, Dopamine release across paired CS+/US trials during conditioning after devaluation. Trial 0 is the first CS+ presentation in the conditioning phase (either CS+ only or CS+/US paired trial), whereas the subsequent trials depict dopamine responses to the CS+ during CS+/US paired trials. Right, Comparing Trials 0 and 2 within feeding treatments shows a rapid restoration of the dopamine CS+ response upon reintroduction to the paired CS+/US contingency for both pellet and chow. *p < 0.05; **p < 0.01; ***p < 0.001; ns, not significant.