Frequency-based heuristics for material perception

Abstract

People often make rapid visual judgments of the properties of surfaces they are going to walk on or touch. How do they do this when the interactions of illumination geometry with 3-D material structure and object shape result in images that inverse optics algorithms cannot resolve without externally imposed constraints? A possibly effective strategy would be to use heuristics based on information that can be gleaned rapidly from retinal images. By using perceptual scaling of a large sample of images, combined with correspondence and canonical correlation analyses, we discovered that material properties, such as roughness, thickness, and undulations, are characterized by specific scales of luminance variations. Using movies, we demonstrate that observers' percepts of these 3-D qualities vary continuously as a function of the relative energy in corresponding 2-D frequency bands. In addition, we show that judgments of roughness, thickness, and undulations are predictably altered by adaptation to dynamic noise at the corresponding scales. These results establish that the scale of local 3-D structure is critical in perceiving material properties, and that relative contrast at particular spatial frequencies is important for perceiving the critical 3-D structure from shading cues, so that cortical mechanisms for estimating material properties could be constructed by combining the parallel outputs of sets of frequency-selective neurons. These results also provide methods for remote sensing of material properties in machine vision, and rapid synthesis, editing and transfer of material properties for computer graphics and animation.

Keywords: adaptation; image statistics; material perception; spatial frequency.

Figure 1

Images of fabrics used in the classification experiment.

Figure 2

Results of the rating experiment. (A) Examples of materials with highest observer consensus for four opponent material property pairs. (B) Strongest associations across material properties. (C) Results of the correspondence analysis. The two axes are the two orthogonal dimension determined by the correspondence analysis. The locations of the properties on these axes are shown in red (FLEX = flexible, WABS = water absorbent, WREP = water repellent). The numbers refer to the positions of the images in Figure 1 numbered row wise starting from the top left.

Figure 3

Comparisons of amplitude spectra for opponent material properties summarized by spatial frequency histograms, and results of nine-level property ranking task for three observers. (Left column) Fabric images with their amplitude spectra. (Center column) Histograms of amplitude distributions across spatial frequencies. The colored parts of the bars indicate the amount by which one image exceeds the other. (Right column) Curves show the median relative energy at different frequencies for images sorted to nine levels, collapsed into three categories (see Appendix A, Figures S1A1–C3 for detailed results). (A) Flat (top) versus undulated (bottom), (B) Thin (top) versus thick (bottom), and (C) Rough (top) versus soft (bottom) fabrics.

Figure 4

Original and manipulated images and their amplitude spectra. The middle column shows the original images, the first and second column show images with increased energy in the frequency bands, and the fourth and fifth column show images with decreased energy. (A) undulation band, (B) thickness band, (C) roughness band (see also Movies 1–3). (D) Transfer of properties between materials by using structures contained in the frequency band from 2–8 cpi.

Figure 5

Images used in the adaptation and distance experiment. The middle column shows the original images. The first and second column depict manipulations of the images with increased energy in the frequency bands, and the fourth and fifth column show manipulations of the images with decreased energy in the frequency bands. ++ and −− indicate the maximal possible increase or decrease without having to correct for out-of-range pixel value in the resulting images; + and − indicate versions of the images intermediate to the original and the maximally changed images.

Figure 6

Experimental sequence and results of the adaptation experiment. (A) Different frequency bands were tested in different blocks. Across blocks, the location of the noise patches and stimuli was alternated between left and right of the fixation point, and above and below. Initial adaptation was 60 s, with 10 s top ups. The stimuli were presented for 0.8 s. (B) Baseline (black) and postadaptation (red) psychometric curves for material property comparisons of two fabrics per frequency band, averaged across five observers. The y axis shows the percentage of trials in which the original image (test stimulus) was seen as being rougher, thicker, and more undulated, respectively, than the images (comparison stimuli) indicated on the x axis. Error bars show ± one SEM.

Figure 7

Experimental setup (A), and results of the distance experiment (B). The results for the different images are shown in separate columns. The x axis indicates the type of comparison stimulus shown on the reference monitor. The y axis shows the percentage of trials in which the test stimulus presented on the test monitor was chosen as being more undulated, thicker, or rougher, respectively, than the comparison stimuli. Colors and symbols indicate the different conditions: both monitors at the same distance (black, circles), test monitor closer to the observer (red, squares), test monitor farther from the observer (blue, triangles). Symbols indicate the mean across three observers. Error bars show ± one SEM.

Figure 8

Results of canonical correlation analysis. (A) Loadings for the independent variable (amplitudes in 37 frequency bands for 161 images). (B) Squared loadings for the independent variable. Error bars denote 95% confidence intervals resulting from 1,000 replications of the canonical correlation analysis. In each bootstrap, a new sample was created by sampling with replacement from the data set.