Novelty encoding by the output neurons of the Basal Ganglia

Abstract

Reinforcement learning models of the basal ganglia have focused on the resemblance of the dopamine signal to the temporal difference error. However the role of the network as a whole is still elusive, in particular whether the output of the basal ganglia encodes only the behavior (actions) or it is part of the valuation process. We trained a monkey extensively on a probabilistic conditional task with seven fractal cues predicting rewarding or aversive outcomes (familiar cues). Then in each recording session we added a cue that the monkey had never seen before (new cue) and recorded from single units in the Substantia Nigra pars reticulata (SNpr) while the monkey was engaged in a task with new cues intermingled within the familiar ones. The monkey learned the association between the new cue and outcome and modified its licking and blinking behavior which became similar to responses to the familiar cues with the same outcome. However, the responses of many SNpr neurons to the new cue exceeded their response to familiar cues even after behavioral learning was completed. This dissociation between behavior and neural activity suggests that the BG output code goes beyond instruction or gating of behavior to encoding of novel cues. Thus, BG output can enable learning at the levels of its target neural networks.

Keywords: primate; reinforcement learning; spikes; substantia nigra pars reticulata.

Figure 1

Task and monkey behavior. (A) A schematic example of the flow of the behavioral task. Cues were followed by an outcome in a probabilistic manner. In each recording session a new cue was randomly interleaved between familiar cues. In this example the right fractal image is a new cue that follows familiar cues (the two left cues). (B) Behavior index (average ± SEM, across behavioral sessions) as a function of the number of new aversive trials. In each session the behavioral response (Lick – Blink) was calculated in a moving average of 20 new trials (and corresponding familiar events). Bin 1 is therefore the average of the first 20 trials, and so on. The responses were normalized between 0 and 1 for each session and then averaged across sessions. Top – new cue is p = 2/3 aversive cue. Bottom- new cue is the p = 1/3 aversive cue, The N in the top-right corner of each plot is the number of recording sessions. (C) Same as (B) for the sessions with new reward cues (Top, p = 2/3; bottom, p = 1/3 new reward cues).

Figure 2

Examples of the activity of two SNpr neurons during presentation of new and familiar cues. (A) The behavior-index during the learning of a new aversive cue as a function of the new and total (new + familiar) number of trials in a single behavioral session. The new cue predicted the aversive outcome with a probability of 2/3. Responses were smoothed by a moving average of 20 new trials (black line) or a moving average of 20 corresponding familiar trials (blue and red lines for the rewarding and aversive events, respectively). Dots on the blue and red lines mark times in which the responses to the new cue were significantly different from the response to the familiar cue (t-test, p < 0.05). (B) Response of a SNpr neuron to familiar and new cues. Spike rate (±SEM shaded) as a function of the new and total number of trials. The cell was recorded at the same time as the behavior shown in (A); the time scales of neural and behavioral changes of this neuron are equal. (C) The peri-stimulus time histogram (PSTH) of the SNpr neuron from (B) during the first twenty trials of recording (top) and the last 20 trials of recording (bottom). PSTHs were constructed by summing activity across trials in a 1 ms resolution aligned at cue presentation (time = 0) and then smoothed with a Gaussian window (SD of 40 ms). (D–F) Same as (A–C) for a different SNpr cell and a different recording session. To enable visualization of the sharp response to some of the events, the PSTH of this cell was smoothed with SD = 20 ms. Unlike the first neuron (A–C), the neural responses of this cell to the familiar cues and new cue are different even after saturation of the behavior.

Figure 3

Monkey's behavior but not SNpr neural activity achieves similarity between responses to new and familiar cues – Population analysis. (A) Fraction of SNpr cells (red) and behavioral sessions (black) which differentiate between responses to a new aversive cue and familiar aversive cues. Data were fitted to an exponential model: Y = A₀ + A₁ · exp(−X / T ); X = number of new trials, weighted according to the number of cells or sessions in each trial. T is the time constant of the exponential fit in units of number of new trials. The fit values appear at the top of the figure; fit values not significantly different from zero (t-test, p > 0.05) are in gray. (B) Same as (A) for the new reward cues. (C) The number of cells used in each bin for the analysis of fraction of cells. After the 60th bin (# new trials 60–79) there was a considerable reduction in the number of recorded cells and hence we limited the analysis in (A) and (B) to the first 60 bins. Note that the increases in the number of cells are due to isolation of new cells during task performance.

Figure 4

The temporal pattern of SNpr neural responses to new cues after saturation of behavioral learning. (A) The PSTHs of the responses to the new cues predicting aversive outcome superimposed for all cells. For this analysis we used only responses after the 50th presentation of the new cue and excluded cells with fewer than 20 new trials. PSTH were smoothed with a Gaussian window with SD = 20 ms and the rate baseline was subtracted to enable comparison between responses. (B) The fraction of SNpr neural responses that showed a significant (2σ rule) increase (blue) or decrease (red) in their discharge rate in response to the new cue after saturation of behavioral learning. (C) The temporal pattern of SNpr encoding of new cues. The fraction of cells in which the response to the new aversive cue was significantly different than the response to familiar aversive events as a function of the time after cue presentation in bins of 200 ms. Black – all responses, blue/red – responses with increase/decrease of discharge rate. (D–F) Same as (A–C) for the familiar and new cues predicting reward outcome.