The effect of atomoxetine on random and directed exploration in humans

Abstract

The adaptive regulation of the trade-off between pursuing a known reward (exploitation) and sampling lesser-known options in search of something better (exploration) is critical for optimal performance. Theory and recent empirical work suggest that humans use at least two strategies for solving this dilemma: a directed strategy in which choices are explicitly biased toward information seeking, and a random strategy in which decision noise leads to exploration by chance. Here we examined the hypothesis that random exploration is governed by the neuromodulatory locus coeruleus-norepinephrine system. We administered atomoxetine, a norepinephrine transporter blocker that increases extracellular levels of norepinephrine throughout the cortex, to 22 healthy human participants in a double-blind crossover design. We examined the effect of treatment on performance in a gambling task designed to produce distinct measures of directed exploration and random exploration. In line with our hypothesis we found an effect of atomoxetine on random, but not directed exploration. However, contrary to expectation, atomoxetine reduced rather than increased random exploration. We offer three potential explanations of our findings, involving the non-linear relationship between tonic NE and cognitive performance, the interaction of atomoxetine with other neuromodulators, and the possibility that atomoxetine affected phasic norepinephrine activity more so than tonic norepinephrine activity.

Fig 1. Task design.

Panel A presents three screenshots showing events surrounding one free choice in a horizon 6 game. The first four choices were constrained to one of the two options in order to control the information available to the subject (A, left). The constraint was indicated by a white box surrounding the choice that the subject was forced to select in order to continue. The columns on each side of the screen showed the history of previous choices, and the number of choices remaining (empty boxes). The example shown represents the unequal information condition, because more is known about the left option than about the one on the right. When a choice was made (A, middle) the outcome value was revealed, and when the next option was presented (A, right) this outcome appeared in the history of the chosen option. Panel B gives a schematic of the different trial types in the three horizon conditions. The first free choice (colored orange) yielded the critical data analyzed here and in previous work [4].

Fig 2. Choice curves as a function of horizon and information condition.

When exploration is motivated by a long horizon, the slope of the curve gets less steep (more decision noise), and the entire curve shifts toward the more informative option (in the unequal information condition), illustrating how the more informative option has value that offsets taking a lower reward in order to explore. Note error bars are 95% confidence intervals.

Fig 3. Subject-level Bayesian estimates of the information bonus (a) and decision noise (b, c) for each treatment.

Both the information bonus and decision noise were markedly increased at longer horizons compared to baseline (horizon 1). Treatment reduces the increase in decision noise from baseline to later horizons in both the equal information condition (b), and the unequal information condition (c) Error bars reflect 95% confidence intervals.

Fig 4. Graphical representation of the model.

Hierarchical Bayesian models yield parameter estimates for each condition at both the subject level and the group level (Lee & Wagenmakers, 2014).