Binding space and time through action

Abstract

Space and time are intimately coupled dimensions in the human brain. Several lines of evidence suggest that space and time are processed by a shared analogue magnitude system. It has been proposed that actions are instrumental in establishing this shared magnitude system. Here we provide evidence in support of this hypothesis, by showing that the interaction between space and time is enhanced when magnitude information is acquired through action. Participants observed increases or decreases in the height of a visual bar (spatial magnitude) while judging whether a simultaneously presented sequence of acoustic tones had accelerated or decelerated (temporal magnitude). In one condition (Action), participants directly controlled the changes in bar height with a hand grip device, whereas in the other (No Action), changes in bar height were externally controlled but matched the spatial/temporal profile of the Action condition. The sign of changes in bar height biased the perceived rate of the tone sequences, where increases in bar height produced apparent increases in tone rate. This effect was amplified when the visual bar was actively controlled in the Action condition, and the strength of the interaction was scaled by the magnitude of the action. Subsequent experiments ruled out that this was simply explained by attentional factors, and additionally showed that a monotonic mapping is also required between grip force and bar height in order to bias the perception of the tones. These data provide support for an instrumental role of action in interfacing spatial and temporal quantities in the brain.

Keywords: ATOM; crossmodal correspondence; magnitudes; temporal estimation.

Figure 1.

(a) Dual task set-up: participants perform gripping actions with increasing/decreasing force (all experiments), or with sustained force (Experiment 5) while simultaneously evaluating whether a tone sequence accelerated or decelerated. A red bar provided a visual indicator of grip force. The upwards or downwards motion of two green triangular trackers provided the required rate of force increase or decrease (or indication of force error in Experiments 4 and 5). Participants had to adjust force throughout the trial to match the bar height to the tracker's motion (Increase/Decrease trials). (b) Both the action and the tone sequence took place in 3900 ms, after which participants responded whether the tone sequence appeared to accelerate or decelerate in a 2AFC scenario. (c) Inter-onset interval of each tone sequence (8 tones, 7 intervals). Δt ms indicates the difference in ms between the first and seventh interval. (d) Experiment 1. In the Action trials, grip force was directly mapped to the changes in bar height (Spatial Magnitude), i.e. increases (or decreases) in force were translated online into increases (or decreases) in bar height. In the No Action trials, the changes in bar height were externally controlled and perfectly matched the motion of the green trackers. (e) Experiment 1—PSI (points of subjective isochrony) in Δt ms for Increase/Decrease trials of the Action and No Action condition (error bars depict standard error). (Online version in colour.)

Figure 2.

(a) Experiment 2. Action trials were identical to those in Experiment 1. In the No Action trials a random jitter was added to the externally controlled increases or decreases in bar height. Simultaneously to monitoring the tone sequence, participants had to indicate if the jittering bar had been on average more time above or below the trackers throughout the trial. (b) Experiment 2—Performance in the No Action-jitter detection task. Proportion of responses ‘bar was more time above the trackers' as a function of trials where the mean bar position was below, centred or above the tracker position. (c) Experiment 2—PSI (points of subjective isochrony) for Increase/Decrease trials of the Action and No Action-jitter task condition. (d) Experiment 3. Participants performed trials involving Large or Small changes in grip force and bar height (Δ G/B). In the Large Δ G/B trials, participants had to match the bar height to the trackers, which moved upwards or downwards across the whole extent of the rectangular frame. In the Small Δ G/B trials, the tracker's motion was confined to a smaller region around the centre of the rectangular frame. (e) Experiment 3—PSI (points of subjective isochrony) for Increase/Decrease trials in the Large and Small Action conditions. (Online version in colour.)

Figure 3.

(a) Experiment 4. Grip force was mapped onto a visual depiction of force error. Participants had to increase or decrease force at an appropriate rate. If the bar was below the trackers, it meant the rate was too slow, while above the trackers it meant the rate was too fast. Ideally participants had to increase or decrease force at a rate that maintained bar height constant. (b) Experiment 4—PSI (points of subjective isochrony) for Increase/Decrease force trials. (c) Experiment 5. Sustained grip force was mapped onto a visual depiction of force error (Sustained Mapping). A constant grip force determined the velocity with which bar height would increase or decrease. Participants had to maintain a target force level to produce increases/decreases in bar height that matched the tracker's motion. If the bar was below the trackers, it meant the force was too low, and vice versa. The Sustained Mapping condition was compared to a Direct Mapping condition (identical to Experiment 1). (d) Experiment 5—PSI (points of subjective isochrony) for Increase/Decrease bar height trials, in the Sustained and Direct Mapping conditions. (e) Experiment 6. Grip force was inversely mapped to bar height where increases (or decreases) in force translated to decreases (or increases) in bar height. The Inverse mapping condition was compared to a Direct mapping condition. (f) PSI (points of subjective isochrony) for Increase/Decrease bar height trials in the Inverse and Direct mapping conditions. (Online version in colour.)