Flies and humans share a motion estimation strategy that exploits natural scene statistics

Abstract

Sighted animals extract motion information from visual scenes by processing spatiotemporal patterns of light falling on the retina. The dominant models for motion estimation exploit intensity correlations only between pairs of points in space and time. Moving natural scenes, however, contain more complex correlations. We found that fly and human visual systems encode the combined direction and contrast polarity of moving edges using triple correlations that enhance motion estimation in natural environments. Both species extracted triple correlations with neural substrates tuned for light or dark edges, and sensitivity to specific triple correlations was retained even as light and dark edge motion signals were combined. Thus, both species separately process light and dark image contrasts to capture motion signatures that can improve estimation accuracy. This convergence argues that statistical structures in natural scenes have greatly affected visual processing, driving a common computational strategy over 500 million years of evolution.

Figure 1

Multiple correlations signify natural image motion. Each row presents a comparison between correlational motion signatures. Columns present: (i) context for each comparison; (ii) properties of pairwise motion estimators; (iii) properties of diverging 3-point estimators; and (iv) properties of converging 3-point estimators. (ai) Motion is approximated by the rigid translation of natural images. (aii-aiv) Cartoon of the correlation structure that each estimator detects. (bi) Example natural image. (bii–biv) Pixelwise contributions to motion estimation are highly variable and differ across estimators. (ci) An ensemble of natural images. (cii–civ) The accuracy with which correlations convey motion is examined across this ensemble. The performance of each estimator is quantified through the Pearson’s correlation between the estimator output and the simulated velocity. We linearly combined estimators to quantify the improvements afforded by multiple correlational signals. The numbers above each bar denote the fractional increase with respect to the 2-point estimate. (d) Same as (c), but with signals spatiotemporally filtered to match motion processing in Drosophila. Error bars are standard deviations over cross-validating trials (see Online Methods).

Figure 2

Triple correlations distinguish between light and dark moving edges. (a) A dark point becomes light when a light edge moves across the visual field (rows 1 and 3), and a light point becomes dark when a dark edge moves across the visual field (rows 2 and 4). We decompose the net pair correlation motion signal into four elements whose frequency of occurrence depends upon the motion. This net pair correlation motion signal reflects the direction of motion (compare rows 1 and 2 to rows 3 and 4) and is insensitive to whether the edge was light or dark (compare row 1 to row 2 or row 3 to row 4). (b) We similarly decompose the net diverging and converging triple correlation into four elements (shown for the diverging triple correlation). The sign of the net diverging triple correlation depends both on the contrast polarity of the edge and on the direction of motion (shown for rightward motion). Thus, triple correlations jointly encode the direction and contrast polarity of a moving edge. (c) Natural motion comprises both moving light edges and moving dark edges. Motion signals are associated with each moving edge, but only the 3-point motion signatures distinguish between edge contrast polarities.

Figure 3

Drosophila responds to triple correlations. (a) Binary spatiotemporal patterns, glider stimuli with 2- and 3-point contrast correlations, were presented to flies. Space-time plots for each of the 6 gliders, and an uncorrelated stimulus, are shown. (b) During the presentation, we measured flies’ turning in response to each glider. Positive rotational velocities represent turning in the direction of the ‘centroid’ of the pattern (to the right in the space-time plots in (a)). (c) One second periods of glider stimuli were interleaved with uncorrelated stimuli; the timing of the presentation of the gliders is denoted by the thick black bar. Response curves show the mean (solid line) and SEM (shading) over flies. (d) Mean turning velocities were computed for each glider by averaging over 0.5s of the stimulus (gray bar in (c)). Turning responses are presented for wild-type Drosophila, alongside the predicted response of a Hassenstein-Reichardt Correlator (HRC) to each glider. N=12 in (c) and (d). ‘**’ denotes a difference from 0 at the p<0.01 level (two-tailed t-test); from right to left, the marked p-values are 4.4×10⁻³ (t₁₁=3.6), 6.0×10⁻⁷ (t₁₁=10.2), 8.2×10⁻⁴ (t₁₁=4.6), and 3.7×10⁻⁶ (t₁₁=8.5). Error bars show SEM.

Figure 4

Detection of triple correlations associated with specific pathways in Drosophila. (a) Left: schematic of the inputs to the fly motion processing pathways. Signals from photoreceptors are relayed through the lamina monopolar cells L1, L2, and L3. Right: a temperature-inducible dominant negative suppressor of synaptic transmission (shi ^ts) was used to silence L1, L2 and L3 using cell-specific expression of Gal4 (L1 shown, red). (b) We examined the responses of these disrupted motion detectors to 3-point gliders. Responses are plotted relative to the 2-point positive glider response. The two control genotypes (Gal4/+ and +/shi ^ts) have all input pathways intact, but contain the genetic constructs for the experimental genotype (Gal4/shi ^ts). For the genotypes Gal4/shi^ts, Gal4/+, +/shi^ts, from top to bottom, N = (19, 14, 19), (18, 13, 19), (29, 16, 19), (22, 14, 19), and (17, 15, 19). Error bars are ± SEM. ‘*’ and ‘**’ represent p<0.01 and p<0.001 differences from both control genotypes (two-tailed t-test).

Figure 5

Triple correlations predict the edge selectivity of motion pathways. (a) The frequency of correlational elements in a moving edge depends on its contrast polarity and direction (see Fig. 2), and we compute the relative abundance of each correlation from the difference in frequency of each element in rightward versus leftward motion (see Fig. S6). The relative abundances of the four triple correlation elements differ between light and dark edges. (b) We used the relative abundance of each correlational element in each edge type (see also Fig. 2, S6) to weight and sum the response of each genotype to each correlational element (Fig. 4). This generated the glider-predicted responses to each edge type, from which we computed the predicted edge selectivity for each genotype. It correlated highly with the behaviorally measured edge selectivity (see Online Methods). Edge selectivity is computed to be the light minus dark edge responses divided by their sum. Error bars on points are ± SEM.

Figure 6

Humans differentially adapt to moving light and dark edges. (a) Schematic of adapter and probe stimulus paradigm (see Figure S4). Black box denotes the time interval used for analysis. (b) Scalp topography of the amplitude of the response at the A′/B′ alternation rate (3 Hz). The amplitude peaks near the occipital pole. (c) Time average of the response from the peak electrode to the probe stimulus under the two adaptation regimes. Response to the unadapted state obtained by probe presentation without the adapting stimulus has been subtracted from this signal (see Figure S4). The response to the probe shows complementary modulation by the adapting stimuli at the frequency of probe alternation (3 Hz). Gray area represents ± 1 SEM. (d) The within subject difference of phase and amplitude at 3 Hz between the two adapting conditions. Ellipse represents 1 SEM while the shaded wedge indicates the 95% confidence interval for the phase. N = 7 subjects in (c) and (d).

Figure 7

Adaptation to moving light and dark edges differentially affects the perception of specific 3-point gliders. Subjects were presented all combinations of four types of adapter stimuli (left column), and 8 gliders (right hand panels), and asked to report the direction of perceived glider motion. Results for each of the 8 glider stimuli are shown, grouped by glider. The color of the bar corresponds to the adapting stimulus: static (black), opposing edges (gray), light edges only (green), or dark edges only (magenta). All stimuli were presented mirror-symmetrically, and responses were aligned to the direction shown in the left hand column. ‘*’, ‘**’, and ‘***’ indicate differences between conditions at p=1.6×10⁻³ (t₁₆=5.6), p=8.8×10⁻⁴ (t₁₄=6.2), and p=2.8×10⁻⁶ (t₁₄=10.2) (two-tailed t-test, Bonferroni corrected for 40 comparisons). N=9 subjects for static and opposing edge adaptation; N=7 for light and dark edge adaptation conditions.