Strategy-dependent effects of working-memory limitations on human perceptual decision-making

Abstract

Deliberative decisions based on an accumulation of evidence over time depend on working memory, and working memory has limitations, but how these limitations affect deliberative decision-making is not understood. We used human psychophysics to assess the impact of working-memory limitations on the fidelity of a continuous decision variable. Participants decided the average location of multiple visual targets. This computed, continuous decision variable degraded with time and capacity in a manner that depended critically on the strategy used to form the decision variable. This dependence reflected whether the decision variable was computed either: (1) immediately upon observing the evidence, and thus stored as a single value in memory; or (2) at the time of the report, and thus stored as multiple values in memory. These results provide important constraints on how the brain computes and maintains temporally dynamic decision variables.

Keywords: computational biology; decision making; human; neuroscience; psychophysics; systems biology; working memory.

Figure 1.. Behavioral task.

Participants were asked to maintain visual fixation on the center cross while an array of colored disks was presented for 0.5 s, followed by a variable delay and finally the presentation of a visual cue whose color was either: (1) the same as one of the disks, indicating that the participant should use the mouse to mark the remembered location of that disk (‘Perceptual’ trial) or (2) white, indicating that the participant should mark the mean angle of the array (‘Computed’ trial). Perceptual and Computed trials were presented in separate, signaled blocks. On Perceptual trials, participants did not know in advance which disk would be probed on any given trial. The number of disks and length of the delay period were varied randomly within each block. Blocks were also defined by the temporal presentation of the disks. In ‘Simultaneous’ blocks, all disks were presented at once, whereas in ‘Sequential’ blocks, the final disk (always the most counter-clockwise of all disks presented on that trial) was presented midway through the variable delay. In all blocks, the disks always had the same clockwise ordering by color, as depicted in the ‘memory array’ graphic above, to minimize binding errors between color and location in the Perceived blocks.

Figure 2.. Diffusion model and predictions for different strategies.

(a) Fifty simulated trials of the representation of a single memorandum, x̂_I, corrupted by a static noise term representing sensory and motor noise (η₁) and time-dependent noise (increasing variance corresponding to decreasing memory precision) modeled as Brownian diffusion. At time t, the report for one item, r_t,₁, is the location of the particle. (b) Linear accumulation of noise (variance) for single or multiple Perceived items (colors, as indicated) or Computed mean values using two different strategies (solid vs. dashed black lines, as indicated). Memory representations of N=1, 2, or 5 items have initial, additive error η_N and diffuse over time with diffusion constant σ_N²; thus, variance at time t=η_N+t*σ_N². For the Average-then-Diffuse (AtD) model, the average is calculated immediately and stored as a single value. Thus, the diffusion constant of a Computed mean of N items is the same as for one item (σ_MN²=σ₁²; parallel purple and black lines), although η₁ and η_MN may not be equal. For the Diffuse-then-Average (DtA) model, all items are stored until the probe time. Thus, the effective σ_MN² is 1/N^th of σ_N². (c) Relationship between A and log differences of diffusion constants for various set sizes and models. σ₁² is independent from A and equal to σ_MN² under AtD. σ_N² is linear with A in log space with respect to (σ₁²) because log(σ_N²)–log(σ₁²)=A*log(N). σ_MN² is linear with A. DC=Diffusion Constant. (d) Accumulation of noise for Perceived items presented sequentially. When the new (Late) point is added at time T/2, the diffusion constant for previously presented items (Early) changes slightly because of the increased load. Early and Late items for set size N have encoding noise η_NE and η_NL, respectively, represented by η_(E/L). The ‘effective Early’ trace shows the net gain in variance over time that would be expected when sampling the error only at a single time T, as we did. (e) Accumulation of noise for Computed items in the Sequential condition for both models. The encoding noise for the mean of N items is represented by η_MNSeq. At time=T/2, the final point is averaged, causing a change in the diffusion constant. The ‘effective’ lines represent the measured change in variance over time one would measure when recording only at T. Here N=5, A=0.5.

Figure 2—figure supplement 1.. Identifiability of AtD and DtA models as a function of the A parameters.

(a) Simulations from set size 2. (b) Simulations from set size 5. For each participant, 1000 task simulations were generated using their best-fit model and parameters, and then each simulation was refit to both the AtD and DtA models and the fits compared via log-likelihoods. The ordinate indicates the percentage of those simulations for which the true (simulate) generative model was identified correctly. In general, identifiability was higher when A was not near 1, as expected, and for set size 5 versus 2. AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 3.. Behavioral summary for the Simultaneous condition.

(a) Mean Perceptual error for different set sizes (colors, as indicated) and delay times (abscissa). Filled points indicate two-tailed t-test for H₀: mean=0, p<0.05. (b) Mean Computed (inferred mean) error for different set sizes (colors, as indicated) and delay times (abscissa). For all tests, mean error was not significantly different from 0 (p>0.05; open circles). (c) Variance in Perceptual errors, plotted as in (a). (d) Variance in Computed (mean) errors, plotted as in (b). In each panel, points and error bars are mean ± SEM across participants.

Figure 3—figure supplement 1.. Full error distributions in Simultaneous conditions.

Each panel shows a histogram of mean error for different delays (colors, as indicated). Perceived trials: (a) set size=1; (b) set size=2; (c) set size=5. Computed trials: (d) set size=2; (e) set size=5. In each panel, points and error bars are mean ± SEM across all participants. Note that in all cases, 95% of the distributions fall between –30° and 30°, justifying our exclusion of larger errors as off-target responses.

Figure 3—figure supplement 2.. Participant-wise mean response error in the Simultaneous Perceived condition.

(a) Delay=0 s, set size=1. (b) Delay=0, set size=2. (c) Delay=0 s, set size=5. (d–f) Same as in (a–c), but for delay of 1 s. (g–i) Same as in (a–c), but for delay of 6 s. In all panels, error bars are ±95% confidence intervals (CIs). Filled points indicate that 0 is not included in the CI, that is, there was a bias in the given participant’s errors.

Figure 3—figure supplement 3.. Participant-wise mean error in the Simultaneous Computed condition.

(a) Delay=0 s, set size=2. (b) Delay of 0, set size=5. (c, d) Same as in (a, b), but for delay of 1 s. (e, f) Same as in (a, b), but for delay of 6 s. In all panels, error bars are ±95% confidence intervals (CIs). Filled points indicate that 0 is not included in the CI, that is, there was a bias in the given participant’s errors.

Figure 4.. Comparisons of empirical and model-based diffusion constant relationships for the Simultaneous condition.

In (a, b, d, e), the abscissa shows the difference between: (1) empirical estimates of the diffusion constant for a Computed value measured by fitting a line to measured variance as a function of delay time for set size 2 (${\displaystyle \hat{\sigma }}$_M2², a, b) or 5 (${\displaystyle \hat{\sigma }}$_M5², d, e), and (2) the empirical estimates of the diffusion constant for a single Perceived value (${\displaystyle \hat{\sigma }}$₁²). The AtD model predicts a difference of 0. The ordinate shows the difference between: (1) the empirical estimate of Computed diffusion constants ${\displaystyle \hat{\sigma }}$_M2² or ${\displaystyle \hat{\sigma }}$_M5², and (2) the empirical estimates of the diffusion constant for multiple Perceived values (${\displaystyle \hat{\sigma }}$₂² or ${\displaystyle \hat{\sigma }}$₅²) divided by the number of items. The DtA model predicts a difference of 0. Each point was obtained using data from individual participants, separated by whether they were best fit by the AtD (a, b) or DtA (d, e) model for the given set-size condition. Lines represent 95% confidence intervals (CIs) computed by simulating data using the best-fit parameters for the given fit and repeating the empirical estimate comparison procedure. Closed symbols indicate participants who fell within the 95% CI for their best-fit model. (e, f) Distance of each participant’s empirically estimated diffusion constant relationships from those predicted by AtD or DtA (i.e., distances from the x=0 and y=0 lines, respectively, in (a, b, d, e)), for set sizes 2 (c) and 5 (f).

Figure 4—figure supplement 1.. Participant-specific estimates of A from the Simultaneous condition for set sizes 2 (a) and 5 (b).

Symbols indicate best-fitting model (AtD or DtA), as indicated. In all panels, error bars are ±95% confidence intervals based on the Hessian computed during model fitting. Note that A=0 implies no difference between the diffusion constant for a single and N items, whereas A=1 implies that the variance and diffusion constant relationship predictions of the AtD and DtA models are equal and thus the models cannot be distinguished from each other. AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 5.. Comparison of model prediction to participant data for the Simultaneous condition.

Each panel shows the empirical variance of participant errors (points and error bars are mean ± SEM data across participants) and model predictions (lines, based on the mean best-fitting parameters across participants for the given model) for the participants’ best fit by the given model (AtD or DtA) for the given condition, as labeled above each column. (a–d) Perceived blocks. (e–h) Computed blocks. AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 6.. Strategy use prevalence in the population.

(a) Difference in log-likelihood between AtD and DtA fits for the Simultaneous condition. Each point represents the difference in fit log-likelihoods for one participant; horizontal bars are medians (solid bar for set size 5 indicates two-sided Wilcoxon signed-rank test for H₀: median=0, p=0.0027). Positive values favor DtA, whereas negative favor AtD. Gray lines connect data generated by the same participant. Only participants whose data were well matched to one of the two models (i.e., within the 95% confidence intervals depicted in Figure 4) were included. (b) Probability of obtaining the proportion of participants’ best fit by each model given average model identifiability of participant parameters. Probability of the results at set size 5 skew toward a higher proportion of AtD users compared to set size 2. AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 6—figure supplement 1.. Relationship between log-likelihood difference for the two strategies and age for the Simultaneous condition.

Log-likelihood comparison for AtD and DtA (negative favors AtD) for set sizes 2 and 5 is not dependent upon age (correlation, ps>0.20 computed separately for each set size). AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 7.. Behavioral summary for the Sequential condition.

(a) Mean error for initially presented (Early) Perceptual items for different set sizes (colors, as indicated) and delay time (abscissa). (b) Mean error for midway presented (Late) Perceptual items for different set sizes (colors, as indicated) and delay time (abscissa). (c) Mean Computed (inferred mean) error for different set sizes (colors, as indicated) and delay time (abscissa). Filled points in (a–c) indicate two-tailed student t-test for H₀: mean=0, p<0.05. (d) Variance in Early Perceptual errors plotted as in (a). (e) Variance in Late Perceptual errors, plotted as in (b). (f) Variance in Computed (mean) errors, plotted as in (c). In each panel, points and error bars are mean ± SEM across participants.

Figure 7—figure supplement 1.. Full error distributions in Sequential conditions.

Each panel shows a histogram of mean error for different delays (colors, as indicated). Perceived Early: (a) set size=2; (b) set size=5. Perceived Late: (c) set size=2; (d) set size=5. Computed trials: (e) set size=2; (f) set size=5. In each panel, points and error bars are mean ± SEM across all participants. Note that in all cases, 95% of the distributions fall between –30° and 30°, justifying our exclusion of larger errors as off-target responses.

Figure 7—figure supplement 2.. Participant-wise mean error in the Sequential Perceived condition.

(a) Delay=1 s, set size=2 for Early samples. (b) Delay=1, set size=5 for Early samples. (c, d) Same as in (a, b), but for delay of 6 s. (e) Delay=0.5 s, set size=2 for Late samples. (f) Delay=0.5 s, set size=5 for Late samples. (g, h) Same as in (e, f), but for delay of 3 s. In all panels, errorbars are ±95% confidence intervals (CIs). Filled points indicate that 0 is not included in the CI, that is, there was a bias in the given participant’s errors.

Figure 7—figure supplement 3.. Participant-wise mean error in the Sequential Computed condition.

(a) Delay=1 s, set size=2. (b) Delay=1, set size=5. (c, d) Same as in (a, b), but for delay of 6 s. In all panels, error bars are ±95% confidence intervals (CIs). Filled points indicate that 0 is not included in the CI, that is, there was a bias in the given participant’s errors (which in this case tended to be toward the mean computed from the early items).

Figure 8.. Comparisons of empirical and model-based diffusion constants.

In (a, b, d, e), the abscissa shows the difference between: (1) empirical estimates of the diffusion constant for a Computed value measured by fitting a line to measured variance as a function of delay time for set size 2 ($\widehat{\sigma }$_M2², a, b) or 5 ($\widehat{\sigma }$_M5², d, e), and (2) the empirical estimates of the diffusion constant for a single Perceived value ($\widehat{\sigma }$₁²) multiplied by the appropriate factor for the set size. The AtD model predicts a difference of 0. The ordinate shows the difference between: (1) the empirical estimate of Computed diffusion constants $\widehat{\sigma }$_M2² or $\widehat{\sigma }$_M5², and (2) the empirical estimates of the diffusion constant of a Computed value based on the DtA hypothesis. The DtA model predicts a difference of 0. Points are data from individual participants, separated by whether they were best fit by the AtD (a, b) or DtA (d, e) model for the given set-size condition. Lines are 95% confidence intervals (CIs) computed by simulating data using the best-fit parameters for the given fit and repeating the empirical estimate comparison procedure. Close symbols indicate participants who fell within the 95% CI for their best-fit model. (e, f) Distance of each participant’s empirically estimated diffusion constant relationships from those predicted by AtD or DtA (i.e., distances from the x=0 and y=0 lines, respectively, in (a, b, d, e)), for set sizes 2 (c) and 5 (f). AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 8—figure supplement 1.. Participant-specific estimates of A from the Sequential condition for set sizes 2 (a) and 5 (b).

Symbols indicate best-fitting model (AtD or DtA), as indicated. In all panels, error bars are ±95% confidence intervals based on the Hessian computed during model fitting. Note that A=0 implies no difference between the diffusion constant for a single and N items, whereas A=1 implies that the variance and diffusion constant relationship predictions of the AtD and DtA models are equal and thus the models cannot be distinguished from each other. AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 9.. Comparison of model fits for the Sequential condition.

Each panel shows the empirical variance of participant errors (points and error bars are mean ± SEM data across participants) and model predictions (lines, using mean predicted variance from each participant’s best-fitting parameters for the given model) for the participants’ best fit by the given model (AtD or DtA) for the given condition, as labeled above each column. (a–d) Errors for Ealy items in Perceived Sequential blocks. (e–h) Errors for Late items in Sequential Perceived blocks. (i–l) Errors for Sequential Computed blocks. AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 10.. Assessment of strategy use prevalence in the population in Sequential conditions.

(a). Difference in log-likelihood per well-fit participant AtD and DtA fits. Negative values favor AtD. Each point represents the difference in fit log-likelihoods for one participant and data from the same participant are connected across set sizes; horizontal bars are medians. Positive values favor DtA, whereas negative values favor AtD. We failed to reject the null hypothesis (two-sided Wilcoxon signed rank test for H₀: median=0, p>0.05) for both set sizes. (b). Probability of obtaining the proportion of participants’ best fit by each model given average model identifiability of each participant parameters. Probability of the results at set sizes 2 and 5 are most likely when the probability of AtD and DtA are similar. AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Figure 10—figure supplement 1.. Relationship between log-likelihood difference for the two strategies and age for the Sequential condition.

Log-likelihood comparison for AtD and DtA (negative favors AtD) for set sizes 2 and 5 is not dependent upon age (correlation, ps>0.20 computed separately for each set size). AtD, Average-then-Diffuse; DtA, Diffuse-then-Average.

Author response image 1.. Results of VBA simulations compared to known ground truth.

a. For the set size 2, Sequential condition, VBA results tended to agree with the ground truth that neither strategy was more frequent (62% of simulations were within 10% of the ground truth). b. For the set size 2, Simultaneous condition, VBA results tended to disagree with the ground truth (only 23% of simulations were within 10% of the ground truth). c. For the set size 5, Simultaneous condition, VBA results tended to agree with the ground truth (90% of simulations were within 10% of the ground truth).