Sensory cue-combination in the context of newly learned categories

Abstract

A large body of prior research has evaluated how humans combine multiple sources of information pertaining to stimuli drawn from continuous dimensions, such as distance or size. These prior studies have repeatedly demonstrated that in these circumstances humans integrate cues in a near-optimal fashion, weighting cues according to their reliability. However, most of our interactions with sensory information are in the context of categories such as objects and phonemes, thereby requiring a solution to the cue combination problem by mapping sensory estimates from continuous dimensions onto task-relevant categories. Previous studies have examined cue combination with natural categories (e.g., phonemes), providing qualitative evidence that human observers utilize information about the distributional properties of task-relevant categories, in addition to sensory information, in such categorical cue combination tasks. In the present study, we created and taught human participants novel audiovisual categories, thus allowing us to quantitatively evaluate participants' integration of sensory and categorical information. Comparing participant behavior to the predictions of a statistically optimal observer that ideally combines all available sources of information, we provide the first evidence, to our knowledge, that human observers combine sensory and category information in a statistically optimal manner.

Figure 1

Cue combination involving categories. A depiction of the problem where each category is defined by two cues. The x and y axes represent the strength of each sensory cue. The circles labelled Y and Z represent the mean and covariance of each cue for categories Y and Z for a given participant, under the assumption that the two cues are conditionally independent. The grey diagonal line represents the linear discriminant vector D onto which an optimal categoriser projects the received bi-cue signal (see text).

Figure 2

Training and Testing stimuli. Black circles represent the occurrence of exemplars of the 2-cue stimuli during training. The elliptical clusters of black symbols represent the Gaussian distributions of the two task-relevant categories. The size of each symbol represents the number of exemplars of each stimulus that were presented during one learning block. Grey squares represent testing stimuli (bimodal in center, unimodal along the x- and y-axes). Twenty-five repetitions of each testing stimulus were presented. Category labels (taygoo and dohkah) and locations (as below or rotated 90°) were counterbalanced across participants.

Figure 3

Cumulative Gaussian fits of unimodal trials for a representative participant. The top left panel plots all five unimodal cumulative Gaussian fits with the PSE equalized to allow for easier slope comparison. The remaining panels plot cumulative Gaussian fits along with data for each unimodal condition separately.

Figure 4

Observed auditory weights for audiovisual trials alongside predictions from the categorical model and the continuous model. Weight predictions for both models are generated including the discount for correlated cues (see main text and equations 6–8). Data points denote means across individual subject weights and error bars denote across-subject standard. Lines are linear fits generated for visualization purposes only.

Figure 5

Trial structure. Training: example of audiovisual training trials with feedback. Testing: example of visual only, audio only, and audiovisual testing trials without feedback.