Dopamine dependence in aggregate feedback learning: A computational cognitive neuroscience approach

Abstract

Procedural learning of skills depends on dopamine-mediated striatal plasticity. Most prior work investigated single stimulus-response procedural learning followed by feedback. However, many skills include several actions that must be performed before feedback is available. A new procedural-learning task is developed in which three independent and successive unsupervised categorization responses receive aggregate feedback indicating either that all three responses were correct, or at least one response was incorrect. Experiment 1 showed superior learning of stimuli in position 3, and that learning in the first two positions was initially compromised, and then recovered. An extensive theoretical analysis that used parameter space partitioning found that a large class of procedural-learning models, which predict propagation of dopamine release from feedback to stimuli, and/or an eligibility trace, fail to fully account for these data. The analysis also suggested that any dopamine released to the second or third stimulus impaired categorization learning in the first and second positions. A second experiment tested and confirmed a novel prediction of this large class of procedural-learning models that if the to-be-learned actions are introduced one-by-one in succession then learning is much better if training begins with the first action (and works forwards) than if it begins with the last action (and works backwards).

Keywords: Computational cognitive neuroscience; Dopamine; Parameter space partitioning; Skill learning; Striatal plasticity.

Figure 1

A. Two sample fractal stimuli used in Experiment 1a. B. Two sample real-world stimuli (indoor scenes) used in Experiment 1b. A. and B. At the start of both experiments 12 stimuli were randomly sampled from a pool of 100 independently for each participant. Four of these 12 stimuli (2 from category A and 2 from category B) were randomly assigned to appear in position 1, another 4 (2As and 2Bs) were randomly assigned to appear in position 2, and the remaining 4 (2As and 2Bs) appeared in position 3.

Figure 2

A. Proportion correct (averaged across participants) from the aggregate-feedback and full-feedback control tasks across blocks in A) Experiment 1a, and B) Experiment 1b. Standard error bars included. (acq: acquisition, agg: aggregate).

Figure 3

A hypothetical example of parameter space partitioning (PSP) for a model with two parameters (a₁ and a₂). Note that in this example, much more of the model’s parameter space is partitioned into the “No learning” than the “Poor Learning” data pattern.

Figure 4

Results of PSP Analysis 1. Percentage of parameter space volume for “None” (red), “Limited” (light green), and “Full” (dark green) learning data patterns, using the immediate-update, feedback-update, and stimulus-feedback-update model versions. Each color corresponds to a unique data pattern discovered by PSP. The height of each colored rectangle corresponds to the volume of parameter space of the specified data pattern.

Figure 5

Results of PSP Analysis 2. Percentage of parameter space volume for “None” (red), “Late” (black), “Early” (light green), and “Throughout” (dark green) learning data patterns, using the immediate-update, feedback-update, and stimulus-feedback-update model versions. The height of each colored rectangle corresponds to the volume of parameter space of that data pattern.

Figure 6

Result of 200 simulations of Experiment 1a by the feedback-update procedural learning model that includes DA release only to the feedback and very low learning rates for positions 1 and 2 (weak eligibility trace, perhaps due to the temporal separation from stimuli to feedback).

Figure 7

Predictions of procedural-learning models in Experiment 2. A) Feedback-update model predictions for 123 training. B) Feedback-update model predictions for 321 training with low learning rates. C) Feedback-update model predictions for 321 training with moderate learning rates. D) Stimulus-feedback-update model predictions for 321 training with high learning rates. (Note. In the legends p1, p2, and p3 signify stimulus positions 1, 2, and 3, respectively.)

Figure 8

Proportion correct (averaged across participants) from Experiment 2 across blocks for A) 123 training and B) 321 training. C) Accuracy to first stimulus presented during 123 training (position 1) and during 321 training (position 3), D) Accuracy to second stimulus presented during 123 and 321 training (position 2 in both cases), and E) Accuracy to third stimulus presented during 123 training (position 3) and during 321 training (position 1). Standard error bars included.