Pharmacological and pupillary evidence for the noradrenergic contribution to reinforcement learning in Parkinson's disease

Abstract

Noradrenaline plays an integral role in learning by optimising behavioural strategies and facilitating choice execution. Testing the noradrenergic framework of learning in the context of human diseases offers a test bed for current normative neuroscience theories and may also indicate therapeutic potential. Parkinson's disease is often considered as a model of dopamine deficits, including dopamine's role in reinforcement learning. However, noradrenergic function is also severely impaired by Parkinson's disease, contributing to cognitive deficits. Using a single dose of the noradrenaline reuptake inhibitor atomoxetine in people with Parkinson's disease (in a randomised double-blind placebo-controlled crossover design), we show improvements in learning compared to placebo. Computational cognitive modelling confirmed a substantial shift in the decision noise parameter, indicative of more exploitative choices. This response pattern closely resembled that of age-matched controls and simulations of optimal response strategies. Pupillometry revealed increased baseline pupil diameter under atomoxetine, which correlated with behavioural improvements, and a heightened phasic pupillary response to feedback. Our findings confirm the noradrenergic contribution to reinforcement learning, and in doing so they challenge the simple interpretation of tonic-phasic locus coeruleus firing patterns based on pupillometry. Noradrenergic modulation is a potential treatment strategy for cognitive symptoms in Parkinson's disease and related disorders.

Fig. 1. Trial structure of the reinforcement learning task and Q-learning model.

a Trial structure: The task involved gain and loss trials, where choosing the correct stimulus resulted in a win of 50 pence versus getting “nil” (gain) or where choosing the correct stimulus resulted in getting “nil” versus losing 50 pence (loss). b Q-learning model: Left panel (learning rate, $\alpha$) shows trajectories of the expected value of a given stimulus-action pair for an example time series of outcomes (green and red dashes), under a range of values of the learning rate; Right panel (inverse temperature, $\beta$) shows the probability of performing a given action as a function of the expected value of that action relative to the expected value of an alternative action, under a range of values of the inverse temperature.

Fig. 2. Reinforcement learning task performance.

a Learning over trials: Smoothed learning curve across the 48 trials (x-axis). Proportion of choices where the winning stimulus was chosen in the gain condition (upper graph; solid line) and where the losing stimulus was avoided in the loss condition (lower graph; dashed line); ribbon represents standard error. b Overall accuracy: Average hit rate, i.e., the proportion of trials where the correct (reward-maximising) stimulus was chosen, plotted separately for each task condition and group/drug condition. Individual participants are represented by dots (n = 19 PD-atomoxetine/PD-placebo; n = 26 controls). Box plot elements: centre line, median; box limits, first and third quartiles; whiskers, most extreme observations within 1.5 times the interquartile range from the box limits.

Fig. 3. Group-level reinforcement learning parameters.

a Group-level reinforcement learning parameters: Estimated posterior distributions of group-level means of learning rate (α) and inverse temperature (β) parameters. b Group-level contrasts: Posterior distributions of group-level means of drug contrasts, i.e., comparison of the distributions between atomoxetine minus placebo. The dark grey sections and percentage annotations denote the proportion of the distribution that was positive. Black dots represent the median and the black line segment represents the 89% highest density interval (HDI).

Fig. 4. Participant-level reinforcement learning parameters.

Point estimates (posterior median) of participant-level learning rate (α; top row) and inverse temperature (β; bottom row) parameters, plotted separately for each task condition (columns) and group/drug condition (colour). Individual participants are represented by dots; grey lines represent within-participant change between placebo and atomoxetine drug conditions (n = 19 PD-atomoxetine/PD-placebo; n = 26 controls). Box plot elements: centre line, median; box limits, first and third quartiles; whiskers, most extreme observations within 1.5 times the interquartile range from the box limits.

Fig. 5. Joint parameter space.

a, d Simulated hit rates for all possible combinations of α and β: Red asterisk denotes the “optimal” parameter combination that yielded the highest simulated hit rate; the contours and colour scale refer to the underlying density distribution of the hit rate estimates, with the contours dividing the range of that data into equally spaced bins; b, e Estimated posterior distributions of group-level means of α and β in the joint parameter space: The two-dimensional density of a given point in the joint parameter space is represented by the colour hue, with darker hues corresponding to greater posterior density. For clarity, points with the lowest observed two-dimensional density (typically corresponding to combinations of α and β that only occurred in a single MCMC sample) are plotted in white, rendering them invisible. c, f Drug effects on the Euclidean distance between estimated and “optimal” joint parameter values: Posterior distributions of the drug effect (i.e., difference between atomoxetine and placebo) on the Euclidean distance between the estimated group-level means of α and β and the optimal α and β. The dark grey sections and percentage annotations denote the proportion of the distribution that was negative (i.e., reduced distance to the optimal values). Black dots represent the median and the black line segment represents the 89% highest density interval (HDI).

Fig. 6. Baseline pupil and pupil temporal derivative during outcome phase of the task.

a Baseline pupil diameter: Median pupil diameter during a 500 ms epoch in the fixation phase, averaged across all trials and task conditions, and plotted separately for each group / drug condition. Individual participants are represented by dots; grey lines represent within-participant change between placebo and atomoxetine drug-state conditions (n = 19 PD-atomoxetine/PD-placebo; n = 26 controls). Box plot elements: centre line, median; box limits, first and third quartiles; whiskers, most extreme observations within 1.5 times the interquartile range from the box limits. b Temporal derivative of pupil signal in the initial 1000 ms window of the outcome phase: Solid lines show group averages of the temporal derivative, and ribbons represent standard error of the mean across trials. Grey bars highlight the group difference in initial phasic response detected by the cluster analysis. Dashed line shows the pupil diameter during this period. The x-axis represents the time (ms) since the start of the outcome phase. c Pupil temporal derivative for early and late task trials: Temporal derivative of pupil signal in the initial 1000 ms window, for the first (early) and second (late) half of the task trials. All visual elements are analogous to those in panel b.

Fig. 7. Associations between atomoxetine effects on baseline pupil and reinforcement learning.

Relationship between atomoxetine-induced change in baseline pupil diameter and loss hit rate (a) and loss β estimates (b). Each dot represents a given Parkinson’s disease participant’s atomoxetine effect (i.e., atomoxetine minus placebo) on the loss hit rate (a) and loss β parameter estimate (b) (y-axis). This captures how their performance changed under atomoxetine, with values above the grey line indicating an increase in values under the drug. The x-axis shows the atomoxetine-induced change in baseline pupil (i.e., atomoxetine minus placebo), with positive values indicating an increase in baseline pupil under the drug. c Atomoxetine-induced change in baseline pupil diameter and loss β estimates, where the thin line elements represent the 89% HDI, thick line elements represent the 50% HDI, and dots represent posterior medians of the atomoxetine effect on the β parameter estimate in the loss condition. d Plausible values analysis of the correlation between baseline pupil and loss β drug effects. Bottom panel: For each MCMC sample, the Pearson correlation between the atomoxetine effects on loss β estimates and baseline pupil diameter was computed, yielding a distribution of plausible correlation coefficients for the current sample of participants (grey histogram). The orange interval plot represents the 89% HDI (thin line), 50% HDI (thick line) and median (point) (n = 19). Top panel: For each of the plausible correlation coefficients, a posterior distribution was computed based on the analytic solution given in Ly et al. (thin grey lines). The mean of these posterior distributions (orange line) provides a summary for inference on the correlation coefficient in the population. The light-orange background represents the 89% HDI of the mean posterior distribution. To avoid overplotting, individual posterior density traces have been plotted for a subset of 500 MCMC samples.