Optimal temporal risk assessment

Abstract

Time is an essential feature of most decisions, because the reward earned from decisions frequently depends on the temporal statistics of the environment (e.g., on whether decisions must be made under deadlines). Accordingly, evolution appears to have favored a mechanism that predicts intervals in the seconds to minutes range with high accuracy on average, but significant variability from trial to trial. Importantly, the subjective sense of time that results is sufficiently imprecise that maximizing rewards in decision-making can require substantial behavioral adjustments (e.g., accumulating less evidence for a decision in order to beat a deadline). Reward maximization in many daily decisions therefore requires optimal temporal risk assessment. Here, we review the temporal decision-making literature, conduct secondary analyses of relevant published datasets, and analyze the results of a new experiment. The paper is organized in three parts. In the first part, we review literature and analyze existing data suggesting that animals take account of their inherent behavioral variability (their "endogenous timing uncertainty") in temporal decision-making. In the second part, we review literature that quantitatively demonstrates nearly optimal temporal risk assessment with sub-second and supra-second intervals using perceptual tasks (with humans and mice) and motor timing tasks (with humans). We supplement this section with original research that tested human and rat performance on a task that requires finding the optimal balance between two time-dependent quantities for reward maximization. This optimal balance in turn depends on the level of timing uncertainty. Corroborating the reviewed literature, humans and rats exhibited nearly optimal temporal risk assessment in this task. In the third section, we discuss the role of timing uncertainty in reward maximization in two-choice perceptual decision-making tasks and review literature that implicates timing uncertainty as an important factor in performance quality. Together, these studies strongly support the hypothesis that animals take normative account of their endogenous timing uncertainty. By incorporating the psychophysics of interval timing into the study of reward maximization, our approach bridges empirical and theoretical gaps between the interval timing and decision-making literatures.

Keywords: decision-making; interval timing; optimality; psychophysics; reward maximization; risk assessment; uncertainty.

Figure 1

Standard normal cumulative distribution functions with the same mean (i.e., 2 s) but different coefficients of variation (CV, σ/μ). Solid curve illustrates the normal cdf for a CV of 0.1 and the dotted curve illustrates the normal cdf for a CV of 0.3. The probability of a random variable taking on a value shorter than the trial time, indicated by vertical solid and dotted lines, respectively, is 0.95. Note that this value is much lower for the simulated subject with smaller timing uncertainty (solid curve).

Figure 2

Expected gain surface (normalized by the maximum expected gain for different levels of timing uncertainty) as a function of target switch latency and the level of timing uncertainty $\left(\widehat{\omega }\right)$. Shades of gray indicate the percentage of normalized maximum expected gain for the corresponding parameter values, $\widehat{t}$ and $\widehat{\omega }.$ (A) is for equally probable short (2 s) and long (3 s) target intervals p(T_S) = 0.5. The ridge of this surface (bold black curve) shows the optimal switch latencies for different levels of timing uncertainty. (B) is for a higher probability of the short target interval, p(T_S) = 0.9. For both cases note the dependence of optimal target switch latencies on the level of timing uncertainty (y-axis). Also note the differences in optimal target switch latencies for two different exogenous uncertainty conditions.

Figure 3

Empirical performance of human and mouse subjects as a function of optimal performance, calculated for the critical task parameters and subjects’ estimated level of endogenous timing uncertainty. Dashed line denotes the identity line. S: subject. Reprinted from Balci et al. (2009). (A) Humans; (B) Mice.

Figure 4

Normalized expected reward rate surface as a function of the normalized target IRT and the level of timing uncertainty. Shades of gray denote the proportion of normalized MPEG, which decreases from light to dark. The solid black curve is the ridge of the normalized expected reward rate surface and denotes the optimal IRT for different levels of timing uncertainty. The dashed vertical black line shows the normalized DRL schedule (in actuality ranging from 5 to 15 s). Each symbol corresponds to the performance of a single rat or human subject in the novel experiment. (A) (Asterisks) shows the human data (DRL 5–15 s), (B) (circles) shows the rat data (DRL 7 s) and (C) (crosses) shows the rat data (DRL 14 s). Data points were clustered near and to the left of the ridge of the expected reward rate surface.

Figure 5

Normalized expected gain surface as a function of normalized aim point and level of timing uncertainty. Shades of gray denote the proportion of normalized MPEG, which decreases from light to dark. The black curve is the ridge of the expected gain surface and denotes the optimal aim points for psychologically plausible levels of endogenous timing uncertainty. Each point (asterisk) corresponds to the performance of a single subject and points are clustered around the curve of the optimal aim points. Figure is redrawn based on the data presented in Simen et al. (2011).

Figure 6

Deviation from the optimal performance curve of the pure DDM as a function of CV redrawn based on the data presented in Balci et al. (2011). Solid line is the linear regression line fit to the data.