Reminders of past choices bias decisions for reward in humans

Abstract

We provide evidence that decisions are made by consulting memories for individual past experiences, and that this process can be biased in favour of past choices using incidental reminders. First, in a standard rewarded choice task, we show that a model that estimates value at decision-time using individual samples of past outcomes fits choices and decision-related neural activity better than a canonical incremental learning model. In a second experiment, we bias this sampling process by incidentally reminding participants of individual past decisions. The next decision after a reminder shows a strong influence of the action taken and value received on the reminded trial. These results provide new empirical support for a decision architecture that relies on samples of individual past choice episodes rather than incrementally averaged rewards in evaluating options and has suggestive implications for the underlying cognitive and neural mechanisms.

Figure 1. Restless bandit task and re-analysis.

(a) Four-armed bandit from Daw et al. (2006). Participants chose between four slot machines to receive points. (b) Payoffs. The mean amount of points paid out by each machine varied slowly over the course of the experiment. (c) Model comparison. Log Bayes factors favouring sampling over the TD model.

Figure 2. Ticket bandit task.

(a) The ticket-bandit task. Each slot machine (‘bandit’) delivered tickets—trial-unique photographs—associated with a dollar value—either −$5 or $5. (b) Payoff probabilities. The probability of each bandit paying out a winning ticket varied slowly over the course of the experiment. Participants were told that their total payout would be contingent both on the number of winning tickets they accrued and their ability to correctly respond on a post-task memory test asking them to recall the reward value and slot machine associated with each ticket. (c) Memory probes. Participants encountered 32 recognition memory probes. On 26 of these probe trials, participants were shown objects that were either received on a previous choice trial (‘valid’), whereas on others they were shown new objects that were not part of any previous trial (‘invalid’). Participants were asked only to perform a simple old/new recognition judgement—to press ‘yes’ if they had seen the image previously in this task and ‘no’ if they had not. After each recognition probe, the sequence of slot machine choices continued as before.

Figure 3. Ticket bandit results.

(a) Model comparison. Log Bayes factors favouring sampling over the TD model. (b) Impact of probes. As in standard RL models, choices are affected by previously observed rewards (black points). Here, memory probes evoking past decisions (red) also modulate choices on the subsequent choice trial. Data points are log odds of choosing the righthand option. (*P<0.05, **P<0.01 and ***P<0.001).