Psychophysical reverse correlation reflects both sensory and decision-making processes

Abstract

Goal-directed behavior depends on both sensory mechanisms that gather information from the outside world and decision-making mechanisms that select appropriate behavior based on that sensory information. Psychophysical reverse correlation is commonly used to quantify how fluctuations of sensory stimuli influence behavior and is generally believed to uncover the spatiotemporal weighting functions of sensory processes. Here we show that reverse correlations also reflect decision-making processes and can deviate significantly from the true sensory filters. Specifically, changes of decision bound and mechanisms of evidence integration systematically alter psychophysical reverse correlations. Similarly, trial-to-trial variability of sensory and motor delays and decision times causes systematic distortions in psychophysical kernels that should not be attributed to sensory mechanisms. We show that ignoring details of the decision-making process results in misinterpretation of reverse correlations, but proper use of these details turns reverse correlation into a powerful method for studying both sensory and decision-making mechanisms.

Fig. 1

Psychophysical reverse correlation has been developed to recover sensory weights in perceptual tasks but it could also be influenced by decision-making mechanisms. a In a typical reverse correlation experiment, subjects receive a sequence of randomly fluctuating sensory information and make a binary choice. Experimenters can directly observe the stimulus and choice, but not the sensory weights and decision-making process (gray box). When choices are made by applying a sensory filter (weighting function) to the stimulus and comparing the result against a criterion, as proposed by signal detection theory, psychophysical reverse correlation will recover the sensory filter. However, it is unknown how well the analysis generalizes to more complex decision-making mechanisms. b Reverse correlation calculates the average stimuli preceding each choice and subtracts the results for the two choices. The outcome is a “psychophysical kernel.”

Fig. 2

The drift diffusion model (DDM) captures the core computations for perceptual decisions made by integration of sensory information over time. We use variants of this model and more sophisticated extensions to explore how the decision-making mechanism influences psychophysical kernels. In DDMs, a weighting function, w(t), is applied to the sensory inputs to generate the momentary evidence, which is integrated over time to form the decision variable (DV). The DV fluctuates over time due to changes in the sensory stimulus and neural noise for stimulus representation and integration. As soon as the DV reaches one of the two decision bounds (+B for choice 1 and −B for choice 2), the integration terminates and a choice is made (decision time). However, reporting the choice happens after a temporal gap due to sensory and motor delays (non-decision time). Experimenters know about the choice after this gap and can measure only the reaction time (the sum of decision and non-decision times) but not the decision time

Fig. 3

Psychophysical kernels deviate from sensory weights in DDM because of incomplete knowledge about decision time. a–c Integration of evidence per se does not preclude accurate recovery of sensory weights. For an unbounded DDM that integrates momentary evidence as long as sensory inputs are available, the kernel matches the true sensory weights. In this simulation, the weight is stationary and fixed at 1, but similarly matching results are obtained for any sensory weight (c; Supplementary Fig. 1). Distortion quantifies root-mean-square error between the psychophysical kernel and the true sensory weights (Eq. 17). d–h The decision bound does not preclude accurate recovery of sensory weights. In a bounded DDM without non-decision time, RTs are identical to decision time (d). Model simulations for RT tasks result in stimulus-aligned kernels that match sensory weights (e) and response-aligned kernels that rise monotonically (f), as expected for termination with bound crossing. However, stimulus-aligned kernels in fixed-duration tasks show a monotonic decrease because later stimuli are less likely to influence the choice (g). This deviation from true sensory weights is caused by early commitment to a choice and becomes smaller as the decision bound rises (h). i–m Variability of non-decision time makes reaction time an unreliable estimate of decision time, causing systematic deviations between psychophysical kernels and true sensory weights. After including non-decision time in the bounded DDM, stimulus-aligned kernels in RT tasks show a monotonic decrease because the stimuli that immediately precede the choice do not contribute to it (j). Response-aligned kernels show a peak, whose time is dependent on the distribution of non-decision times (k). Kernels for fixed-duration tasks are not affected by non-decision time (l) but still show the decline caused by bound crossing, similar to g. Deviation of stimulus-aligned kernels in the RT task increases with variability of non-decision time (m). Standard deviation of non-decision time is assumed to be 1/3 of its mean in these simulations. All kernels are normalized according to Eq. 2 or Eq. 14 to allow direct comparison with the true sensory weights (see Methods)

Fig. 4

Psychophysical kernels in the direction discrimination task match predictions of a bounded DDM with non-decision time. a RT task design. Subjects initiated each trial by fixating on a central fixation point. Two targets appeared after a short delay, followed by the random dot stimulus. When ready, subjects indicated their perceived motion direction with a saccadic eye movement to a choice target. The net motion strength (coherence) varied from trial to trial, but also fluctuated within trials due to the stochastic nature of the stimulus. b, c Choice accuracy increased and RTs decreased with motion strength. Data points are averages across 13 subjects. Accuracy for 0% motion coherence is 0.5 by design and therefore not shown. Gray lines are fits of a bounded DDM with non-decision time. Error bars denote s.e.m. across subjects. d Motion energy of example 0% coherence trials (dotted lines), and the average (solid black line) and standard deviation (shading) of motion energy across all 0% coherence trials. Positive and negative motion energies indicate the two opposite motion directions in the task. e, f The bounded DDM predicts psychophysical kernels (gray lines), which accurately match the dynamics of subjects’ kernels (red lines). Because the model sensory weights are stationary, kernel dynamics in the model are caused by the decision-making process and non-decision times. Kernels are calculated for 0% coherence trials. Shading indicates s.e.m. across subjects. All kernels are shown up to the minimum of the median RTs across subjects

Fig. 5

Psychophysical reverse correlation in a face discrimination task with multiple informative features reveals relative weighting of features and kernel dynamics similar to the direction discrimination task. a Task design. Subjects viewed a sequence of faces interleaved with masks and reported whether the face identity matched one of two prototypes. They reported their choice with a saccadic eye movement to one of the two targets, as soon as ready. b Using a custom algorithm, we designed intermediate morph images between the two prototype faces such that only three facial features (eyes, nose, and mouth) could be informative. These features were morphed independently from one prototype (+100% morph) to another (−100% morph), enabling us to create stimuli in which different features could be biased toward different identities. All regions outside the three informative features were set to halfway between the prototypes and were uninformative. c The three informative features underwent subliminal fluctuations within each trial (updated with 106.7-ms interval). The mean morph levels of the three features were similar but varied across trials. Fluctuations of the three features were independent (Gaussian distribution with standard deviations set to 20% morph level). d, e Choice accuracy increased and RTs decreased with stimulus strength. Data points are averages across nine subjects. Error bars are s.e.m. across subjects. Gray lines are model fits. f The DDM used to fit subjects’ choices and RTs extends the model in Fig. 2 by assuming different sensitivity for the three informative features. Momentary evidence is a weighted average of three features where the weights correspond to the sensitivity parameters. The momentary evidence is integrated toward a decision bound. g Psychophysical kernels estimated from the model (gray lines) match subjects’ kernels for the three features. Shaded areas are s.e.m. across subjects

Fig. 6

Psychophysical kernels are susceptible to changes of decision bound, input correlation, mutual inhibition, integration time constant, and limited dynamic range. The figure shows extensions of DDM and systematic deviations that additional realism to the model can cause in psychophysical kernels. Conventions are similar to Fig. 3, except that we focus only on RT tasks. Also, to isolate the effects of different model parameters from the effect of non-decision time, we use zero non-decision time in these simulations. a–c Collapsing decision bound (urgency signal) inflates the psychophysical kernel over time. The rate of bound collapse is defined by τ_1/2—the time it takes to have a 50% drop in bound height. d–f Extending DDM to a competition between two bounded accumulators reveals that input correlation of the accumulators has only modest effects on psychophysical kernels, causing an initial overshoot followed by an undershoot compared to true sensory weights. g–i The presence of a lower reflective bound in the accumulators causes an opposite distortion: an initial undershoot followed by a later overshoot. j–l Balancing the effect of mutual inhibition by making the integrators leaky causes the model to behave like a DDM, eliminating the effects of both the inhibition and leak on the psychophysical kernels (black curves in m). Any imbalance between leak and inhibition, however, causes systematic deviations in the kernels from the true sensory weights (brown, red, and blue curves in k). See Supplementary Fig. 8 for more examples

Fig. 7

A decision-making model that has a mixture of parameters with opposing effects on psychophysical kernels can create a diversity of kernel dynamics for static sensory weights. a A model composed of two competing integrators that allows different ratios of leak and inhibition, collapsing decision bounds, and non-decision times. The model also has input correlation >−1 and reflective lower bounds, but they are fixed for simplicity. b When bound collapse is small and non-decision times are long, the kernel drops monotonically over time. c When bound collapse is large and non-decision times are short, the kernel rises monotonically. d When these opposing factors balance each other, the kernel becomes flat