Multistability, cross-modal binding and the additivity of conjoined grouping principles

Abstract

We present a sceptical view of multimodal multistability--drawing most of our examples from the relation between audition and vision. We begin by summarizing some of the principal ways in which audio-visual binding takes place. We review the evidence that unambiguous stimulation in one modality may affect the perception of a multistable stimulus in another modality. Cross-modal influences of one multistable stimulus on the multistability of another are different: they have occurred only in speech perception. We then argue that the strongest relation between perceptual organization in vision and perceptual organization in audition is likely to be by way of analogous Gestalt laws. We conclude with some general observations about multimodality.

Figure 1.

Köhler's demonstration of binding by common embodiment. (a) The maluma figure. (b) The takete figure.

Figure 2.

Examples of coherent and unrelated ambiguities: the Adams & Haire [21] nested cubes. (a) Same orientations. (b) Opposite orientations.

Figure 3.

An example of unrelated ambiguities: Ambiguity 1: is the Necker cube interpreted as a cube seen from a vantage point above to its right or from a vantage point below to its left? Ambiguity 2: is the paper cutout of a Necker cube seen floating in front of eight black disks painted on a white background (illustrated in figure 3b) or is it seen through eight holes in a white surface, against a black backdrop? (illustrated in figure 3d and rendered in figure 3c) (a) The Bradley & Petry [22] cube. (b) Default interpretation: modal completion. (c) The modified Bradley & Petry [22] cube. (d) Alternative interpretation: amodal completion).

Figure 4.

Two examples of grouping. (a) Visual. (b) Auditory.

Figure 5.

Theory of indispensable attributes: the visual thought-experiment. (a) Start. (b) Collapse over wavelength: not indispensable. (c) Collapse over space: indispensable.

Figure 6.

Theory of indispensable attributes: the auditory thought-experiment. (a) Start. (b) Collapse over space: not indispensable. (c) Collapse over frequency: indispensable.

Figure 7.

Four-tone bistable tone sequence used by Cook & Van Valkenburg [33]. (a) The middle tones are heard grouped with the high tone, leaving the low tone perceptually isolated. (b) The middle tones are heard grouped with the low tone, leaving the high tone perceptually isolated.

Figure 8.

Wertheimer's [44] rectangular dimotif dot lattices apply grouping by proximity and grouping by similarity concurrently to the same stimuli. (a) Wertheimer's figure (xii): both proximity and similarity favour columns (in terms defined in figure 9, |b|/|a| =1.083). (b) Wertheimer's figure (xiii): proximity favours columns (in terms defined in figure 9, |b|/|a| = 1.104) but similarity favours rows.

Figure 9.

Dot lattices. (a) Defining features. (b) Two-dimensional space and nomenclature.

Figure 10.

The attraction function of grouping by proximity.

Figure 11.

Two dimotif dot lattices. In both, grouping by proximity favours a, but more weakly in the dot lattice on left, where |a| = 1.25|b|, than in the dot lattice on the right, where |a| = 1.5|b|. In the dot lattice on the left, grouping by similarity favours b3 (δ > 0, where δ is a measure of dissimilarity between two kinds of dots), whereas in the dot lattice on the right it favours a (δ < 0), but because the differences between dot-colours are smaller on the left than on the right, the strength of the grouping by similarity (δ = 2) on the left is smaller than the strength of grouping by similarity on the right (δ =− 3).

Figure 12.

A schematic of the results obtained by Kubovy & van den Berg [54] using the dimotif lattices described in figure 11, showing that the conjoined effects of proximity and similarity are additive when choice probabilities are represented as log-odds. The line for δ = 0 (light grey) corresponds to the attraction function in figure 10, i.e. all the dots have the same colour, and therefore, grouping by similarity cannot affect the results. The results are equivalent to the multiplicative model of figure 13.

Figure 13.

Multiplicative model of conjoined grouping that implies additivity of grouping by proximity and grouping by similarity when choice probabilities are represented as log-odds, as in figure 12.

Figure 14.

An auditory necklace 11100110 of length n = 8; i.e. it is eight beats long. It has five notes and three rests, grouped into two sequences of notes called runs, and two sequences of rests called gaps.

Figure 15.

The response screen. At the moment depicted, the cross is highlighted.

Figure 16.

p(r_A ) as a function Δ r and Δ g. Error bars span ± 1 s.e.