Do humans produce the speed-accuracy trade-off that maximizes reward rate?

Abstract

In this paper we investigate trade-offs between speed and accuracy that are produced by humans when confronted with a sequence of choices between two alternatives. We assume that the choice process is described by the drift diffusion model, in which the speed-accuracy trade-off is primarily controlled by the value of the decision threshold. We test the hypothesis that participants choose the decision threshold that maximizes reward rate, defined as an average number of rewards per unit of time. In particular, we test four predictions derived on the basis of this hypothesis in two behavioural experiments. The data from all participants of our experiments provide support only for some of the predictions, and on average the participants are slower and more accurate than predicted by reward rate maximization. However, when we limit our analysis to subgroups of 30-50% of participants who earned the highest overall rewards, all the predictions are satisfied by the data. This suggests that a substantial subset of participants do select decision thresholds that maximize reward rate. We also discuss possible reasons why the remaining participants select thresholds higher than optimal, including the possibility that participants optimize a combination of reward rate and accuracy or that they compensate for the influence of timing uncertainty, or both.

Figure 1

Optimal performance curves. Horizontal axes show the error rate and vertical axes show the decision time (DT) normalized by total delay D_total. The thick line (identical in both panels) is the optimal performance curve for the reward rate (RR). The thin lines show the generalized optimal performance curves for reward accuracy (panel a) and modified RR (panel b). Each curve corresponds to a different value of q ranging from 0 (when it reduces to the optimal performance curve for RR, shown in the thick lines) to 0.5 (top curves) in steps of 0.1.

Figure 2

Average error rate (ER) and reaction time (RT) in units of seconds for all twenty participants of Experiment 1 and all delay conditions. Delay conditions are indicated on x-axes – labels correspond to conditions (1) D=0.5s; (2) D=1s; (3) D=2s; and (4) D=0.5s, D_p=1.5s. Error bars consider comparisons between two adjacent conditions (inspired by Masson & Loftus, 2003). For example, in panel a, error bar for condition D=0.5 and the left error bar for condition D=1 consider comparison between these two conditions. The height of the error bars corresponds to standard error of the differences between ERs (or RTs in panel b) in the two conditions. Specifically, the difference is first calculated for each participant, and then the standard error of the differences is calculated across participants; both error bars are equal to this standard error, so they have equal heights. These error bars have standard interpretation: if two adjacent bars are different from each other by more than approximately two heights of corresponding error bars, then ERs (or RTs) are significantly different according to the paired t-test.

Figure 3

Quantile probability plots showing the fits of the pure (a) and the extended (b) drift diffusion model (DDM) to behavioural data from a sample participant. Open circles correspond to the behavioural data. The horizontal axes indicate the probability of error in the left parts of the panels, and probability of correct choice in the right parts (see labels on x-axes). In each part, the four columns of circles correspond to the four delay conditions in the experiment. In each column of circles, the vertical positions of the five circles indicate the quantiles of reaction times. Small filled circles visualise the corresponding predictions for the DDM. These are connected by lines to make the patterns they create more visible. For clarity, error bars with confidence intervals for quantiles of reaction times are not shown here. They are plotted for the same participant in Figure 5 of Bogacz et al. (2006), which shows that the confidence intervals are very large for the error trials (up to 1.48s for the 0.9 quantile in D=0.5, D_p=1.5 condition) because of the small number of such trials. The estimated parameters of the models (for noise parameter fixed at c=0.1): a) pure DDM: T₀=0.346, A=0.219, z₁=0.0398, z₂=0.0535, z₃=0.0610, z₄=0.0682; b) extended DDM: T₀= 0.372, s_t=0.084, m_A=0.344, s_A=0.152, s_x=0.044, z₁=0.0503, z₂=0.0592, z₃=0.0687, z₄=0.0816.

Figure 4

Comparison of parameters T₀, ã and z̃ estimated by fitting pure (vertical axes) and extended (horizontal axes) drift diffusion model. In the left and the central panels dots correspond to individual participants; in the right panel different experimental delay conditions are indicated by different symbols (see legend).

Figure 5

Comparison of participants' and optimal thresholds in Experiment 1. The left column of panels (a, c, e) compares estimates and predictions of the pure drift diffusion model (DDM), while the right column of panels (b, d, f) – of the extended DDM. In panels a-d, horizontal axes correspond to participants' normalized thresholds, the optimal normalized thresholds are along the vertical axes, and the dotted line is the identity line. Different experimental delay conditions are indicated by different symbols (see legend). Note different scales between rows of panels. Panels a and b show a sample participant with the best fit; panels c and d show data of all 20 participants. Panels e and f show the mean participants' (black bars) and optimal (white bars) thresholds averaged across participants for different delay conditions (indicated on the horizontal axis). Error bars show standard error (there are error bars on bars corresponding to optimal thresholds, because different participants have different optimal thresholds). Stars indicate the level of significance of the difference between participants' and optimal thresholds (paired t-test): one star denotes p < 0.05, two stars denote p < 0.01, three stars denote p < 0.001.

Figure 6

Mean error rate and mean reaction time (in seconds) for all 60 participants in Experiment 2 for each delay condition averaged across difficulty conditions. Error bars as in Figure 2.

Figure 7

Comparison of participants' and optimal thresholds in Experiment 2. Left panels (a and c) show findings for the easy condition, right panels (b and d) for the difficult condition. In panels a and b, participants' normalized thresholds are shown on horizontal axes, optimal normalized thresholds on the vertical axes, and the dotted line is the identity line. Different experimental delay conditions are indicated by different shapes (see legend). Note different scales. Panels c and d show the mean participants' (black bars) and optimal (white bars) thresholds averaged across participants for different delay conditions (indicated on the horizontal axis). Error bars show standard error, and stars indicate the level of significance of the difference between participants' and optimal thresholds (paired t-test): one star denotes p < 0.05, two stars denote p < 0.01, three stars denote p < 0.001.

Figure 8

Performance of participants with reward scores higher than the median reward score for Experiment 2. a) Relationship between theory match error (x-axis) and achieved reward score (y-axis). Each symbol corresponds to one participant; genders as indicated in key. b, c) Mean error rate and reaction time (in units of seconds) averaged across difficulty conditions. Error bars as in Figure 2. d, e) mean participants' (black bars) and optimal (white bars) normalized thresholds averaged across participants for different delay conditions (indicated on the horizontal axis). Panel d shows data for easy conditions; panel e – for difficult conditions. Error bars show standard error, and stars indicate the level of significance of the difference between participants' and optimal thresholds (paired t-test): one star denotes p < 0.05, two stars denote p < 0.01, three stars denote p < 0.001.

Figure 9

Optimal performance curves. In all panels except b, horizontal axes show the error rate (ER) and vertical axes show the normalized decision time (DT), i.e. DT divided by total delay D_total. Note the differences in scales between panels. Solid black curves show the theoretical prediction based on the assumption that participants maximize the reward rate. a) Normalized DTs as a function for ER plotted for one participant of Experiment 2. Each circle corresponds to one experimental condition. For two conditions the normalized DTs and ERs were very close, so two circles overlap (in the bottom right corner of the panel). b) Histogram of the correlations between normalized DT and the normalized DT predicted by the optimal performance curve. Each correlation is calculated for one participant of Experiment 2. Shaded bars indicate the correlations that were statistically significant (p<0.05). In panels c-e, the bars show the mean normalized DT averaged across participants' conditions with ERs falling into the given interval (from Experiments 1 and 2). The error bars indicate standard error. There are no error bars for some bins, as there was just one condition falling into this bin, or DTs for participants' conditions falling into this bin were all very close to 0. c) The bars are based on all participants. d) The participants are divided into three groups on the basis of their reward scores and bars in each of the three panels are based on the data from the corresponding group of participants (see titles of the panels). e) Data from conditions with D_p=0 based on subgroups of participants as in panel d. Best fitting optimal performance curves for RA and RR_m are also shown (see legend).

Figure 10

Changes in participants' performance within single blocks of trials. In all panels horizontal axes show the trial number within a block – each point corresponds to a bin of 5 trials. Vertical axis shows the difference in the mean reaction time (RT) between delay condition D = 2, and D = 0.5 averaged over: 5 trials in a given bin and both difficulty conditions. Error bars indicate standard error. Panel a shows data averaged across participants of Experiments 1 and 2 with reward scores higher than the median for each of experiment, and panel b shows data averaged across participants with reward scores lower than the median.