Neural signature of fictive learning signals in a sequential investment task

Abstract

Reinforcement learning models now provide principled guides for a wide range of reward learning experiments in animals and humans. One key learning (error) signal in these models is experiential and reports ongoing temporal differences between expected and experienced reward. However, these same abstract learning models also accommodate the existence of another class of learning signal that takes the form of a fictive error encoding ongoing differences between experienced returns and returns that "could-have-been-experienced" if decisions had been different. These observations suggest the hypothesis that, for all real-world learning tasks, one should expect the presence of both experiential and fictive learning signals. Motivated by this possibility, we used a sequential investment game and fMRI to probe ongoing brain responses to both experiential and fictive learning signals generated throughout the game. Using a large cohort of subjects (n = 54), we report that fictive learning signals strongly predict changes in subjects' investment behavior and correlate with fMRI signals measured in dopaminoceptive structures known to be involved in valuation and choice.

Fig. 1.

Schematic of the idea of a fictive error and task design. (A) At time t, an agent in state S_t transitions to a new state S_t+1 by taking action a_t and observes a reward r_t. However, the agent also observes other rewards r′_t that could have been received had alternative actions a′_t been chosen. (B) The figure under time t − 1 shows the state of the task immediately after a snippet of market has been revealed. At time t the subject makes a new allocation between cash and stock (in this case increasing the bet). When the market goes up, bigger investments are immediately revealed as better choices, generating the fictive error “best choice − actual choice.” Likewise for market drops, smaller investments would have been better.

Fig. 2.

Experiment screen and time line. (A) Screen like that seen by subject (the background was dark in the scanner). The subject has just lost 23.92% (right box), has a portfolio worth $139 (left box), has 50% invested in the market (middle bar), and has nine choices remaining (from examining the screen). (B) Time-line of experiment. After the market outcome is revealed, the middle bar (which indicates the bet size) is grayed out, and a new bet cannot be submitted. The bar is illuminated 750 ms later, and the subject has a free response period to submit a new bet. After the new bet is submitted, the next snippet of market is revealed 750 ms later. The major regressors (including the fictive error) used in the fMRI analysis are time-locked to this event.

Fig. 3.

Influence of fictive error signal on behavior. Barplot of the average normalized change in next investment versus the level of the fictive error. Changes in investment were converted into z-scores within each subject. The fictive error signal was binned into three levels [(0.00, 0.04), (0.04, 0.08), (0.08, ∞)] for the figure (see SI Fig. 8 for a scatterplot). Error bars are standard errors.

Fig. 4.

Brain responses to fictive error signal. (Upper) SPM t-statistic map for the fictive error regressor (f_t⁺) showing activation in motor strip (a), inferior frontal gyrus (b), caudate and putamen (c), and PPC (d). Threshold: p < 1 × 10⁻⁵ (uncorrected); cluster size ≥5. Slices defined by y = 8 and y = −72. Random effects over subjects, n = 54. (Lower) SPM t-statistic map for the positive market return (r_NL⁺) regressor in the “Not Live” condition showing no activation in the striatum but strong activation in the visual cortex (e). Threshold: p < 1 × 10⁻⁵ (uncorrected); cluster size ≥5. Slices defined by y = 8 and y = −72. Random effects over subjects, n = 54.

Fig. 5.

Basic TD regressor and fictive error signal. SPM t-statistic maps of the basic TD regressor (Upper) and the fictive error signal (Lower) showing activation in the striatum associated with each. The fictive error regressor is orthogonalized with respect to the TD regressor. Threshold: p < 1 × 10⁻⁵ (uncorrected); cluster size ≥5. Random effects over subjects, n = 54. (Insets) Separate colored-coded activations for fictive error only, TD error only, and the overlap region of the two. These activations are shown at three levels of significance and suggest that activations to fictive error only may be segregated to the ventral caudate.

Fig. 6.

Q-learning TD regressor and fictive error signal. SPM2 t-maps of the Q-learning TD regressor (Upper) and the fictive error signal (Lower) again showing activation in the ventral striatum associated with the TD error, and in the ventral caudate for the fictive error. The fictive error regressor is orthogonalized with respect to the TD regressor. Threshold: p < 1 × 10⁻⁵ (uncorrected); cluster size ≥5. Random effects over subjects, n = 54. (Insets) Separate colored-coded activations for fictive error only, TD error only, and the overlap region of the two. The area of overlap is larger for the Q-learning model and fictive error than for the TD regressor and fictive error (see Fig. 5).