Can monkeys choose optimally when faced with noisy stimuli and unequal rewards?

Abstract

We review the leaky competing accumulator model for two-alternative forced-choice decisions with cued responses, and propose extensions to account for the influence of unequal rewards. Assuming that stimulus information is integrated until the cue to respond arrives and that firing rates of stimulus-selective neurons remain well within physiological bounds, the model reduces to an Ornstein-Uhlenbeck (OU) process that yields explicit expressions for the psychometric function that describes accuracy. From these we compute strategies that optimize the rewards expected over blocks of trials administered with mixed difficulty and reward contingencies. The psychometric function is characterized by two parameters: its midpoint slope, which quantifies a subject's ability to extract signal from noise, and its shift, which measures the bias applied to account for unequal rewards. We fit these to data from two monkeys performing the moving dots task with mixed coherences and reward schedules. We find that their behaviors averaged over multiple sessions are close to optimal, with shifts erring in the direction of smaller penalties. We propose two methods for biasing the OU process to produce such shifts.

Figure 1. The motion discrimination task.

Target colors cue the magnitude of rewards for correct responses, red denoting a value twice that of blue. The four panels in the reward segment show the possible reward conditions. See text for full description.

Figure 2. A typical state space of the LCA model, showing nullclines on which for (thin curves), fixed points (filled circles with arrows indicating stability types) and slow manifold (dashed line).

Diagonal solid line represents one-dimensional state space of reduced OU model, with associated probability distribution of sample paths.

Figure 3. Psychometric functions showing fraction of T1 choices as a function of coherence for constant reward bias applied before and during motion period.

(A) ; (B) ; (C) ; each panel shows the cases and −0.1 (left to right). Remaining parameters are and (arbitrary time units). Green lines indicate slopes for zero bias; arrows show shifts.

Figure 4. Optimal shifts as a function of the reward ratio r ₁/r ₂ for fixed coherences (solid blue curves) and for coherence ranges centered on the fixed coherences (dashed red curves).

(A):  = 10; 20 and 30% (top left to bottom right, solid blue), and [C ₁;C ₂] = [5; 15]; [15; 25] and [25; 35] (top left to bottom right, dashed red). (B): Coherence bands centered on  = 20% (solid blue curve) with widths 10; 20; 30 and 40% (bottom left to top right, dashed red). Approximation of Eq. (30) shown in green. The slope b ₁ is fixed at 0.06 throughout.

Figure 5. Fits of accuracy data from monkeys A (A) and T (B) to the PMF (15), for the four reward conditions averaged over all sessions.

Bars denote standard errors. See text for details.

Figure 6. Optimal shifts b ₂ for a range of reward ratios r ₁/r ₂ and b ₁ = 0.0508 (solid, black) and b ₁ = 0.0432 (dot-dashed, red), corresponding to slopes of PMFs fitted to equal rewards data for monkeys A and T.

Vertical dotted lines at r ₁/r ₂ = 0.5 and 2 intersect the curves at the symmetrically-placed optimal shifts for those reward ratios. (A) Predictions for the different sets of nonuniformly-distributed coherences viewed by each animal. (B) Results for coherences distributed uniformly from −48% to 48%: note smaller optimal shifts and reversal of order of curves for A and T compared to panel A. Triangles and crosses respectively indicate shifts determined from data for monkeys A and T for r ₁/r ₂ = 0.5, 1 and 2 (cf. Table 1).

Figure 7. Contours (black curves) of expected rewards for for monkeys A (A) and T (B) over the ()-plane, based on the coherences viewed by each animal.

Vertical dashed lines indicate values fitted to pooled equal rewards data. Note that gradients in in either direction away from ridges of maximum expected rewards (blue curves) become smaller as decreases, that gradients are smaller for overshifts in than for undershifts, that this asymmetry increases as decreases, and that gradients are steeper for T than for A. See text for discussion.

Figure 8. Optimal PMFs (black curves) and bands (color) in which 99.5% of maximal possible rewards are gained, compared with session-averaged HL, LL and HH, and LH data (triangles, left to right on each panel) for monkeys A (A) and T (B).

See text for details.

Figure 9. Slope and shift values for individual sessions and the four reward conditions, plotted as points in the for monkeys A (four panels in (A)) and T (four panels in (B)).

Asterisks indicate values averaged over all sessions (cf. top two rows of Table 1). Performance curves and bands show optimal values for given values (central blue curves) and values that gain 99% and 97% of maximum rewards are also shown (flanking magenta curves closest to and farthest from blue curves, respectively).