Expectations affect the perception of material properties

Abstract

Many objects that we encounter have typical material qualities: spoons are hard, pillows are soft, and Jell-O dessert is wobbly. Over a lifetime of experiences, strong associations between an object and its typical material properties may be formed, and these associations not only include how glossy, rough, or pink an object is, but also how it behaves under force: we expect knocked over vases to shatter, popped bike tires to deflate, and gooey grilled cheese to hang between two slices of bread when pulled apart. Here we ask how such rich visual priors affect the visual perception of material qualities and present a particularly striking example of expectation violation. In a cue conflict design, we pair computer-rendered familiar objects with surprising material behaviors (a linen curtain shattering, a porcelain teacup wrinkling, etc.) and find that material qualities are not solely estimated from the object's kinematics (i.e., its physical [atypical] motion while shattering, wrinkling, wobbling etc.); rather, material appearance is sometimes "pulled" toward the "native" motion, shape, and optical properties that are associated with this object. Our results, in addition to patterns we find in response time data, suggest that visual priors about materials can set up high-level expectations about complex future states of an object and show how these priors modulate material appearance.

Figure 1.

Contribution of prior associations and image cues on perceived material qualities. The role of predictions or associative mechanisms in material perception is not well understood. (A) The perception of material qualities (such as gelatinousness) can be influenced by prior associations between dynamic optics, shape, and motion properties. (A) Watching the green (left) object deform may evoke an association with green Jell-O (B), and may therefore be perceived as wobblier and more gelatinous than the matte object, despite both objects wobbling in identical ways (as shown in Supplementary Movies S1 and S2). (C) Alternatively, the green object may be perceived as wobblier owing to larger image differences between frames, and potentially higher motion energy, as illustrated on the right (Doerschner et al., 2011, Doerschner, Kersten, & Schrater, 2011). A combination of associative and modulatory mechanisms is also possible. The difference in motion energy in images of the translucent object in (C) is about seven (6.8) times larger than that of the matte one, purely owing to the difference in optical properties between these two objects.

Figure 2.

Three frames from “Preposterous” by Florent Porta. Artists have played with our expectation of how objects and their materials should behave. In this study, we compare material perception for falling objects that deform in surprising and unsurprising (i.e., expected) ways. Retrieved from https://vimeo.com/191444383.

Figure 3.

Stimuli expected motion. Shown are all 15 familiar (top) and corresponding novel objects (bottom) used in the experiments. Objects are organized according to their typical material kinematics, that is, how they deform under force. Note that individual scenes are scaled to maximize the view of the object (first frame, left columns), or to give a good impression of the material kinematics (last frame, right columns).

Figure 4.

Trial and stimuli. (A) An exemplar trial. (B, C) A subset of familiar objects (B) and corresponding novel objects (C) used in the experiments. Familiar objects could either behave as expected, or in a surprising manner. Note that this distinction (expected vs surprising) is only meaningful for familiar objects. Note that individual scenes are scaled to maximize the view of the object (first frame), or to give a good impression of the material kinematics (last frame). Figure 3 (expected condition) and Supplementary Figure 1 (surprising condition) show corresponding views for the entire stimulus set. The objects were rendered at approximately the same size as each other, so that for example the key was the same “physical” (simulated) size as the chair, even though in real life chairs are larger than keys. We chose to do this so that the objects would hit the ground at the same time, and behave in a similar way under gravity. This is important for some of our analyses (see Analysis—Response time).

Figure 5.

Ratings results from all experiments. Shown are average observer ratings of all experiments for the four different questions about material qualities, for example, how hard, liquid, heavy, or gelatinous an object appears. Each column shows the data across experiments for one particular type of material kinematics (wobble, splash, etc.). Icons symbolize individual familiar objects (chair, key, cup, pot, glass, spoon; blue droplet, water; yellow droplet, honey; white droplet, milk; violet curtain, silk; red curtain, velvet; white curtain, linen; yellow custard, red and green Jell-O). Each rating question and corresponding data are coded in the same color (red, How hard?; blue, How liquid?; purple, How heavy?; and green, How gelatinous?). Ratings could vary between 0 (lowest) and 1 (highest). The circle and star symbols correspond with ratings of familiar and novel objects, respectively. The symbol style—filled, desaturated, and open—correspond with the different experiments—that is, first frame, last frame, and motion, respectively. Standard deviations denote 1 SE of the mean. Overall, ratings of familiar and novel objects tended to overlap much more in the motion condition (C, E), in particular the expected motion condition (C), than compared with the first frame (A) and last frame (B, D) experiments. In the surprising conditions, the object identity and how the object deforms after falling onto the floor mismatch (e.g., shattering water, splashing chairs, etc.). Data from the first frame (A) and motion conditions (C, E) are replotted in Figure 7 to illustrate the influence of prior expectation on material quality judgements.

Figure 6.

Effect of the expectation index (∈). Shown are the averages across participants, material types and rating attributes for familiar and novel objects. ∈ was calculated as the absolute value of the average rating differences between expected and surprising conditions (see analysis section, Equation 1). Error bars are one standard error of the mean.

Figure 7.

Material quality ratings and prior attraction. Results of the first frame and motion conditions from Figure 5 are replotted, keeping the same symbols and notation. (A) Average observer ratings from three conditions (i.e., first frame familiar objects [filled circles], typically behaving familiar objects [unfilled circles] and corresponding [moving] novel objects [unfilled stars]) tended to overlap. The difference between first frame ratings for familiar objects and ratings of moving novel objects is indicated by a dark grey line. The organization of objects follows that in (B). (B) Same as (A), but here, ratings of atypically behaving familiar objects are plotted as unfilled circles (organized by type of motion), and ratings of corresponding novel objects—that is, unfamiliar shapes—which inherit their optical and kinematic qualities from a familiar object—as unfilled stars. The motions are arranged according to how much the object remains intact and recognizable after impact on the floor (also see Supplementary Fig. 4). Yellow highlighted symbols show a statistically significant prior pull. See main text for more detail. The yellow highlighted cases show that prior pull occurred more in conditions where the object was still intact and recognizable at the end of the movie (objects that behaved rigidly or wobbled). Supplementary Table 1 lists corresponding statistics and P values. (C, i) How we measure how much the rating of an atypically moving familiar object (middle) overlaps with the rating of a material-matched moving novel object (right), or conversely, how much it is pulled toward ratings of a static view of the familiar object (left). (C, ii) Possible results. For example, seeing an image of red Jell-O in its classical shape, observers tend to expect that it is quite gelatinous. When they see an object with the same optical properties that falls and does not wobble when it hits the floor, they rate it as very nongelatinous, that is, we have a large rating difference (gray line). When a classically shaped red Jell-O falls on the floor and does not wobble, observers could either rate it similar to the novel object—after all it does not wobble at all (no prior pull)—or it could be rated as somewhat more gelatinous, despite the sensory input, possibly because prior experience influences the appearance, making observers perceive wobble when there is not (prior pull, red line). (C, iii) When the familiar object moves exactly as expected, and when there is no strong influence of shape familiarity on material judgements, all three ratings will overlap.

Figure 8.

Prediction strength response time differences and interobserver correlations. (A) Correlation between mean first frame ratings and mean expected motion ratings for familiar and novel objects. (B) A high correlation indicates that the first frames (still images) of objects are highly predictive of the objects’ kinematic properties, and thus are in good agreement with ratings in the expected motion condition, where objects fall and deform according to their typical material kinematics. This is clearly not the case for novel objects, suggesting that these objects do not elicit strong prior expectations about how an object will deform. (C) Average interobserver correlation for expected and surprising motion trials, as well as the first frame experiment. Note that only for novel objects, this latter correlation was quite low, suggesting that still images of unfamiliar objects do not elicit a strong prior in observers about the material qualities measured in this experiment. (D) Response time data averaged across all observers for expected (black) and surprising trials (medium gray). Stars indicate significant differences, P < .001. Error bars are 1 standard error and show variability between subjects.

Figure 9.

Low and high level linear regression models predicting the difference between ratings of moving familiar and control objects D_{familiar-novel}. We developed two models with the aim to account for the differences we observed in ratings of familiar and novel moving objects. The computation of the individual predictors is described in the main text (Analyses section). In the lower right inset of each plot we show the weights (w) of each predictor (H1: bounded shape and optics prior; H2, shape familiarity; H3, last frame shape recognizability; L1, motion energy difference [also see Supplementary Figure 6]; L2, object size differences first frame; L3, object size differences last frame). Overall, the high-level model was more successful in predicting this difference than the low-level model (top two panels: combined). However, this pattern varied as a function of rating questions: the high-level model performed best for ratings of gelatinousness and hardness, whereas the low-level model performed as good as the high-level model for ratings of liquidness. The latter is likely due to the fact that ratings of liquidness might be strongly modulated by how much a substance physically spreads in the image.