The extraction of 3D shape from texture and shading in the human brain

Abstract

We used functional magnetic resonance imaging to investigate the human cortical areas involved in processing 3-dimensional (3D) shape from texture (SfT) and shading. The stimuli included monocular images of randomly shaped 3D surfaces and a wide variety of 2-dimensional (2D) controls. The results of both passive and active experiments reveal that the extraction of 3D SfT involves the bilateral caudal inferior temporal gyrus (caudal ITG), lateral occipital sulcus (LOS) and several bilateral sites along the intraparietal sulcus. These areas are largely consistent with those involved in the processing of 3D shape from motion and stereo. The experiments also demonstrate, however, that the analysis of 3D shape from shading is primarily restricted to the caudal ITG areas. Additional results from psychophysical experiments reveal that this difference in neuronal substrate cannot be explained by a difference in strength between the 2 cues. These results underscore the importance of the posterior part of the lateral occipital complex for the extraction of visual 3D shape information from all depth cues, and they suggest strongly that the importance of shading is diminished relative to other cues for the analysis of 3D shape in parietal regions.

Figure 1.

Visual stimuli of the main experiments. (A) Texture stimuli: 3D lattice (black), 3D constrained (gray), constrained scrambled (light green), lattice scrambled (dark green), uniform texture (dark blue); lattice aligned (light blue). (B) Shading stimuli: 3D shaded (black), center shaded (orange), shaded blob (dark red), unshaded blob (light red), uniform luminance (dark yellow), and pixel scrambled (light yellow). (C) Parts of image with aligned dots (red rectangle) and patch of uniform dots (green square) in 3D lattice stimulus.

Figure 2.

Luminance histograms and amplitude spectra of the stimuli used in the main experiment. (A) Luminance histograms averaged over all 11 shapes (the error bars indicate standard deviations [SDs]) of 3D shaded, center-shaded, and shaded-blob stimuli. Yellow bars indicate the luminance of the uniform-luminance stimuli and the arrows indicate the light and dark gray values for each of the unshaded-blob shapes (see Materials and Methods). (B) Amplitude spectra averaged over the 11 shapes (the error bars indicate SDs). Upper panel: spectra of 3D shaded (black line), center-shaded (orange line), shaded-blob (dark red), and unshaded-blob (light red) shapes; lower panel: spectra of 3D shaded (black line), uniform luminance (yellow line), and pixel scrambled (olive line). The stars indicate frequencies at which spectra differed significantly (1-way ANOVA, P < 0.05) between 3D shaded stimuli and the 2D controls (orange: center shaded; olive: pixel scrambled). Notice that the amplitude spectra were not calculated on the interior of the shapes but on the central 15.4° x 14.5° part of the display.

Figure 3.

Average luminance distribution across images in the 6 conditions of the SfS experiment. (A) The luminance distributions in the central 14.5° x 14.5° part of the display, averaged across the 11 stimuli used in each condition. (B) luminance plotted as a function of vertical position to highlight the upper–lower asymmetries in the stimuli. The 4 lines (color code see inset) correspond to 2.5° wide vertical strips, centered on the fixation point, over which luminance was integrated.

Figure 4.

Results from psychophysical experiments 1 and 2. Depth magnitude estimation task in complete population of subjects (n = 28 in the shading and n = 12 in the texture experiments, A) and the 6 subjects participating in experiment 2 (B), and 3D shape adjustment task (C). Apparent depth (A, B) and average setting of 8 points (C), averaged over stimuli and subjects, is plotted as function of conditions. Error bars indicate SEs across participants. In (C) arrows indicate ground truth.

Figure 5.

3D SfS and 3D SfT sensitive regions rendered on the fiducial brain. Shading and texture specific areas shown on the posterior parts of left and right hemispheres (ventral-lateral view) of the fiducial PALS atlas (Van Essen 2005). Green (A) and yellow (B) patches represent voxels significantly (P < 0.001 uncorrected, Conjunction null analysis, random effect, n = 18) activated by 3D shape defined by texture and shading, respectively. The local maxima are indicated with black (texture) and red (shading) circles (filled circles: significance at P < 0.05 corrected; open circles: significance P < 0.0001 uncorrected), labeled with the numbers used in Table 1: (1) and (2) L and R caudal ITG, (3) and (4) L and R LOS, (5) L VIPSt, (6) R VIPS, (7) R POIPSt, (8) R DIPSM, (9) L DIPSM, and (10) R DIPSAs.

Figure 6.

3D SfS and 3D SfT sensitive regions plotted on the flattened hemispheres (posterior part). Green patches and yellow outlines represent voxels significantly (P < 0.001 uncorrected, Conjunction null analysis, random effect, n = 18) activated by 3D shape defined by texture and shading, respectively. Same conventions as in Figure 5. The 2D shape-sensitive regions (i.e., voxels significant in subtraction intact versus scrambled images) from the LOC localizer (Denys et al. 2004; Kourtzi and Kanwisher 2000) are indicated by dashed orange outlines. Motion-sensitive areas are outlined with blue dashed lines and the blue stars indicate the motion-sensitive region hMT/V5+ (Sunaert et al. 1999). The threshold for LOC and motion localizers is set to P < 0.0001 uncorrected (n = 18, random effect). The early retinotopic areas V1, V2 and V3 (from Caret atlas) are indicated by dotted white lines. White dots indicate location of the V1 probes used in Figure 11 and Table S4. R, right; L, left; CS, calcarine sulcus; STS, superior temporal sulcus; ITS, inferior temporal sulcus; OTS, occipital temporal sulcus; CollS, collateral sulcus.

Figure 7.

3D SfS and 3D SfT sensitive regions (single-subject analysis, same conjunction analysis as in Figs 5 and 6). Statistical parametric maps, indicating voxels significantly (P < 0.001 uncorrected) active when viewing 3D shapes defined by shading (yellow patches) and 3D objects defined by texture (green patches) shown on the flattened surface representation of the left and right hemispheres (posterior part) of 2 of the subjects (A.C. and J.G.). Purple voxels indicate overlap between shading- and texture-sensitive regions. Dashed orange and blue outlines correspond to LOC and motion localizers at threshold P < 0.05 corrected. Solid and dashed white lines indicate the projection of horizontal and vertical meridians in a given hemisphere (Claeys et al. 2004; Fize et al. 2003). For other conventions see Figure 6.

Figure 8.

Activity profiles plotting averaged MR signal changes compared with fixation condition over all subjects (n = 18). (A) 3D SfT sensitive regions: L and R caudal ITG, L and R LOS, L VIPSt, R VIPS, R POIPSt, L and R DIPSM, and R DIPSAs. (B) 3D SfS sensitive regions: L and R caudal ITG. The error bars indicate SEs between subjects. Color bars indicate different conditions following the same convention as in Figure 1.

Figure 9.

Comparison of activation patterns in conjunction and individual contrasts of the SfT experiment on flattened hemispheres. SPM plotting the voxels significant at the P < 0.001 unc level (random effects) in the conjunction (A), the contrast 3D constrained minus 2D constrained-scrambled (B), 3D lattice minus 2D lattice scrambled (C), 3D lattice minus 2D lattice-aligned (D), and 3D lattice minus 2D uniform texture (E). In (B–E) the red outlines reproduce the activation in the conjunction (A).

Figure 10.

Comparison of activation patterns in conjunction and individual contrasts of the SfS experiment on flattened hemispheres. SPM plotting the voxels significant at the P < 0.001 unc level (random effects) in the conjunction (A), the contrast 3D shaded minus 2D center shaded (B), 3D shaded minus 2D shaded blob (C), 3D shaded minus 2D unshaded blob (D), 3D shaded minus 2D pixel scrambled (E), and 3D shaded minus 2D uniform luminance (F). In (B–F) the red outlines reproduce the activation in the conjunction (A).

Figure 11.

Activity profiles, averaged across subjects (n = 18) and local maxima, of the 2D shape-sensitive areas and dorsal and ventral V1 in the main experiments: (A) 3D SfT and (B) 3D SfS. Mid-FG (LOa), post-ITG (LO), LOS and DIPSMs (Denys et al. 2004; Kourtzi and Kanwisher 2000) were defined by the subtraction intact versus scrambled images (at P < 0.0001 uncorrected random effect). In 4 subjects V1 was defined retinotopically and the averaged coordinates were applied across all subjects. Error bars indicate SEs between subjects. The locations of the V1 probes are indicated in Figure 6. Same color conventions as in Figure 1.

Figure 12.

(A and C) Activity profiles plotting percent MR signal change compared with fixation condition in the main 3D SfS sensitive regions (L and R caudal ITG) when the same visual stimuli were presented with Lambertian (A) and specular (C) shading surfaces (fixed effect, n = 4). (B and D) Examples of a 3D shape with, respectively, Lambertian (B) and specular (D) shading.

Figure 13.

Results of psychophysical experiments 4 and 5. (A) Slope of line relating texture settings and shading settings in the 3D shape adjustment task plotted as a function of depth magnitude (in % of original). Individual values (black) and medians (red) are plotted. (B and C) Mean and SE of d′ and reaction time in the same-different task plotted as a function of the amplitude of the shape perturbation. In (A) the median correlations were 0.28, 0.95, and 0.98, respectively.

Figure 14.

3D SfT sensitive regions (control experiment 2, fixed effect, n = 6). (A) Texture specific areas significantly (P < 0.05 corrected) activated during passive viewing (dark blue voxels, light blue overlap with “active” voxels) and when the participants performed 1-back task (green voxels) shown on the PALS flattened representations of left and right hemispheres (posterior part). The significant local maxima defined by the active runs are indicated with red dots (1) R V3v, (2) and (3) L and R caudal ITG, (4) and (5) L and R mid-FG (middle fusiform gyrus), (6) L mid CollS (collateral sulcus), (7) R mid LG (lingual gyrus), (8) and (9) L and R LOS, (10), and (11) L and R VIPS, (12) and (13) L and R POIPS, (14) and (15) L and R SPL (superior parietal lobule), (16) R DIPSM, (17) L DIPSA, (18) L phAIP (Binkofski et al. 1999; Grefkes et al. 2002), and (19) R post CS (central sulcus) (Table S2). (B and C) Activity profiles of the 19 local maxima in the passive (fixation) and active (1-back task) runs, respectively. Black stars indicate the areas significant in passive conjunction analysis (see Materials and Methods). Yellow stars indicate areas significant in the interaction: (uniform luminance minus fixation) task − (uniform luminance minus fixation) passive. Inset: examples of the visual stimuli used in the experiment.

Figure 15.

3D SfS sensitive regions (control experiment 2, fixed effect, n = 6). (A) Shading specific areas significantly (P < 0.05 corrected) activated during passive viewing (yellow voxels) and when the participants performed 1-back task (red voxels) shown on the PALS flattened representations of left and right hemispheres (posterior part). The yellow dotted outlines indicate voxels significant at P < 0.001 uncorrected from the passive-viewing experiment. These yellow outlines and voxels are contained in the “active” red voxels. The significant local maxima defined from the active runs are indicated by green dots: (1) and (2) L and R caudal ITG, (3) and (4) L and R mid-FG (middle fusiform gyrus), (5) and (6) L and R mid CollS (collateral sulcus), (7) R mid LG (lingual gyrus), (8.1, 8.2) and (9) L and R LOS, (10) L VIPS, (11) L DIPSM, (12) L DIPSA (Table S3). For other conventions see Figure 6. (B and C) Activity profiles of all local maxima in the passive (fixation) and active (1-back task) runs. Black stars indicate the areas significant in passive conjunction analysis (see Materials and Methods). Yellow stars indicate areas significant in the interaction: (uniform luminance − fixation) task − (uniform luminance − fixation) passive. Inset: examples of the visual stimuli used in the experiment.