Multisensory causal inference is feature-specific, not object-based

Abstract

Multisensory integration depends on causal inference about the sensory signals. We tested whether implicit causal-inference judgements pertain to entire objects or focus on task-relevant object features. Participants in our study judged virtual visual, haptic and visual-haptic surfaces with respect to two features-slant and roughness-against an internal standard in a two-alternative forced-choice task. Modelling of participants' responses revealed that the degree to which their perceptual judgements were based on integrated visual-haptic information varied unsystematically across features. For example, a perceived mismatch between visual and haptic roughness would not deter the observer from integrating visual and haptic slant. These results indicate that participants based their perceptual judgements on a feature-specific selection of information, suggesting that multisensory causal inference proceeds not at the object level but at the level of single object features. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Keywords: causal inference; cue integration; roughness; slant; visual–haptic.

Figure 1.

Set-up and stimuli. (a) Participants viewed stereoscopically presented visual stimuli via a mirror so that they were perceived as co-located with virtual haptic stimuli rendered using a Phantom force-feedback device. (b) The stimuli were rough surfaces, slanted top–back from fronto-parallel. Participants were trained to haptically explore the surfaces following a sinusoidal path illustrated in red. (c) A red occluder was placed in front of the rough surfaces to limit geometric cues for surface slant. In visual and visual–haptic conditions, a peephole in the centre of the occluder opened once participants touched the virtual stimulus. Participants wore active shutter glasses so that separate images could be presented to either eye (here, the image presented to the left eye is placed at the right side to enable crossed fusion). (Online version in colour.)

Figure 2.

Results. (a) Psychometric curves for two participants (one per row) in the visual–haptic condition of the roughness (left column) and slant (right column) tasks. Markers indicate the observed proportion of ‘more rough / more slanted than the standard' responses for each feature level shown on a common scale for roughness and slant. Grey dashed curves show psychometric curves fitted to these data; red curves show psychometric curves corresponding to maximal integration effects given the participant's performance in unimodal trials (see electronic supplementary material, S1 for all participants and all conditions). Shaded ribbons indicate 95% confidence intervals for both curves. Top row: sample participant who showed maximal integration effects for roughness but not for slant. Bottom row: sample participant who showed the reversed pattern. (b) Integration indices for both features and all participants. The integration index is the ratio of the standard deviation of the fitted visual–haptic curve and the predicted curve assuming maximal integration effects (see electronic supplementary material, S3 for an alternative index). An index of one indicates maximal integration effects, while larger values suggest less-than-maximal integration effects, indicating perceptual judgements that are partially based on unimodal information. Error bars indicate 95% confidence intervals obtained by bootstrapping the raw data. (Online version in colour.)

Figure 3.

Model predictions. (a) Distribution of simulated correlation coefficients between the integration indices for roughness and slant. Correlation coefficients are based on 10 000 simulated datasets of the same size as the original data (26 participants, 20 trials per condition). Data were generated using the feature-specific (brown) and object-based (orange) causal-inference models (see Methods, §4f). Vertical lines indicate distribution means. (b) Visual–haptic psychometric curves (black dashed lines) for a single simulated observer with modality-specific biases for roughness but not slant. The feature-specific causal-inference model (top row) predicts a clear deviation from optimal cue integration (solid red lines)—i.e. less than maximal integration effects—for roughness but not for slant, whereas the object-based (bottom row) causal-inference model predicts reduced integration effects for both features. (Online version in colour.)