Inferring the stiffness of unfamiliar objects from optical, shape, and motion cues

Abstract

Visually inferring the stiffness of objects is important for many tasks but is challenging because, unlike optical properties (e.g., gloss), mechanical properties do not directly affect image values. Stiffness must be inferred either (a) by recognizing materials and recalling their properties (associative approach) or (b) from shape and motion cues when the material is deformed (estimation approach). Here, we investigated interactions between these two inference types. Participants viewed renderings of unfamiliar shapes with 28 materials (e.g., nickel, wax, cork). In Experiment 1, they viewed nondeformed, static versions of the objects and rated 11 material attributes (e.g., soft, fragile, heavy). The results confirm that the optical materials elicited a wide range of apparent properties. In Experiment 2, using a blue plastic material with intermediate apparent softness, the objects were subjected to physical simulations of 12 shape-transforming processes (e.g., twisting, crushing, stretching). Participants rated softness and extent of deformation. Both correlated with the physical magnitude of deformation. Experiment 3 combined variations in optical cues with shape cues. We find that optical cues completely dominate. Experiment 4 included the entire motion sequence of the deformation, yielding significant contributions of optical as well as motion cues. Our findings suggest participants integrate shape, motion, and optical cues to infer stiffness, with optical cues playing a major role for our range of stimuli.

Introduction

The ability to identify materials and estimate their properties by sight is invaluable for many tasks, from selecting ripe fruit to avoiding icy patches when walking. Humans are highly adept at visually identifying and categorizing materials, allowing inferences about possible uses and predicted behavior of these materials and the objects made from them. Indeed, it has been argued that successful behavior depends as much on perceiving materials as on perceiving objects (Adelson, 2001; Anderson, 2011; Fleming, 2014). Consequently, the visual perception of material classes (e.g., Fleming, Wiebel, & Gegenfurtner, 2013; Sharan, Rosenholtz, & Adelson, 2014; Wiebel, Valsecchi, & Gegenfurtner, 2013, 2014) as well as the visual estimation of specific properties of materials (such as glossiness; e.g., Kim, Marlow, & Anderson, 2012; for a review, see Chadwick & Kentridge, 2015) has received increasing attention in recent years. Interestingly, there have not been many studies on how inferences about objects are driven by the interplay between their perceived material and their shape, although previous studies suggested object shape to be the primary source for visual identification, categorization, and prediction of future behaviors of objects (e.g., Aliaga, O'Sullivan, Gutierrez, & Tamstorf, 2015; Biederman, 1987; Landau, Smith, & Jones, 1988; van Assen & Fleming, 2016; Paulun, Schmidt, van Assen, & Fleming, 2017). 

Here, we study this interplay for a material property that has been investigated mostly in haptics: stiffness (or its inverse compliance), which determines how much an object will deform in response to an applied force. Like other physical properties, stiffness provides many important cues to usability and behavior of objects, for example, whether an object will deform or break under pressure, whether a loaf is fresh or stale, or even whether an object will evoke feelings of attachment and positive emotion (Harlow, 1958). 

Although stiffness information is mainly obtained from tactile and proprioceptive modalities (e.g., Srinivasan & LaMotte, 1995; Tan, Durlach, Beauregard, & Srinivasan, 1995), human observers can also infer stiffness from visual cues. There are several potential information-processing routes to achieving this (Paulun et al., 2017). 

First, observers could infer stiffness from optical material cues based on previous knowledge about material properties (e.g., “marble is hard”). This would represent an associative approach to inferring unseen physical properties, which relies on correlations between optical and mechanical cues for real-world materials. Findings in the field of material perception suggest that observers are fast at categorizing different materials (Sharan et al., 2014; Wiebel et al., 2014) and consistently label material categories with specific attributes (Baumgartner, Wiebel, & Gegenfurtner, 2013; Fleming et al., 2013). They can distinguish between soft (fabric, foliage, paper) and hard materials (metal, stone, wood; e.g., Fleming et al., 2013), and this distinction in turn influences other processes in visual perception (such as amodal completion; Vrins, Wit, & van Lier, 2009). This suggests that even though optical and mechanical properties are physically independent from one another, learned associations can support judgments about stiffness for familiar materials. 

Second, observers could infer stiffness by identifying image features that are directly influenced by stiffness. This would be an estimation approach based on the causal effects of stiffness on measurable image quantities. In particular, when an object is compliant, it responds to external forces by deforming in distinctive ways, leading to shape and motion cues related to stiffness. Nevertheless, such an approach is computationally challenging: The observed shape and motion depend not only on the intrinsic physical properties of the object but also on the extrinsic forces applied to the object. Disentangling the relative contribution of intrinsic and extrinsic causes is far from trivial. 

Although a number of studies on haptic perception of stiffness included visual or visuo-haptic conditions in which an object or finger indents, displaces, or deforms the surface of a soft body (e.g., Cellini, Kaim, & Drewing, 2013; Drewing & Kruse, 2014; for an overview, see Klatzky & Wu, 2014), the role of more complex shape deformations for the visual perception of stiffness has rarely been studied. Previous studies showed that observers can infer transformations of an object from its shape (Kubilius, Bracci, & Op de Beeck, 2016; Leyton, 1989; Schmidt & Fleming, 2016; Spröte & Fleming, 2015). This inference potentially also affects judgments about the material properties of the object: Bent objects might, for example, be perceived as soft because it was possible to bend them. Moreover, haptic experience of interacting with objects affects the perception of object motion and mental imagery (White, 2012). Here, we wanted to test to what extent participants base their judgments of stiffness on complex shape cues versus optical material cues, in static objects as well as in dynamic animations of shape transformations unfolding over time. 

Previous studies testing the effect of shape cues on stiffness perception often used simple scenarios in which objects of different stiffness (and sometimes different material appearance) dropped onto a rigid ground. For example, Han and Keyser (2015) asked participants to judge the stiffness of free-falling simple geometric shapes (e.g., cylinders), which deformed when hitting the ground. They found that the perceived stiffness in these animated scenes is determined by the available shape deformation cues, whereas different optical material appearances (e.g., metal gas tanks versus sausages) hardly affected stiffness perception. Material merely played a role by modulating the extent to which the shape deformation could be perceived (e.g., materials with rich texture show more details of the deformation; therefore, objects were perceived as softer). Han and Keyser (2016) support this notion by demonstrating that dropping balls were perceived as softer when their checkerboard texture (defined by contrast and spatial frequency) allowed for more details of the shape deformation to be seen. Consequently, both studies assign a rather small weight to optical material cues while showing that shape cues are relatively strong in determining perceived stiffness. 

In a similar paradigm, in which participants also rated the stiffness of a free-falling transparent cube, Kawabe and Nishida (2016) showed that differences in stiffness could be estimated from shape contour deformation alone (even though contour stimuli were generally rated softer compared with the full renderings of the cubes). This was true even when the deformations had to be inferred from the motion of a random noise field. Kawabe and Nishida (2016) also demonstrated the role of motion for the perceived stiffness: when shape information was removed by showing only the internal motion of the cube, observers were still able to infer the different stiffness levels from the different motion patterns. Finally, perceived stiffness increased with increasing animation speed (also see the related work on inference of liquid characteristics from image motion speed and smoothness of motion flow; Kawabe, Maruya, Fleming, & Nishida, 2015). 

Schmid and Doerschner (2017) showed participants free-falling cubes of different substances (soft, medium, and hard), which resulted in dramatic differences in shape deformation (e.g., cracking apart or shattering). Again, they did not find any effects of optical material (here, opaque versus semitransparent versus transparent) on perceived stiffness, even though they found interactions of material and substance of the cubes when asking for other attributes (e.g., heaviness, smoothness; for some attributes, they even report interactions when testing static images, e.g., for wetness, wobbliness). 

Unlike the previous studies investigating objects that deform when falling onto a plane, Paulun et al. (2017) studied stiffness perception for objects rigidly attached to the plane and investigated the potential contribution of optical material cues. They showed observers animated scenes in which a rigid cylinder interacted with cubes of varying stiffness and optical appearance, either (a) by pushing into the cubes from above (i.e., indenting the cubes) or (b) by retracting from the rear edge of the cubes (i.e., setting the cubes into reverberating motion). In line with the findings by Kawabe and Nishida (2016), perceived stiffness varied along with the deformation of the cubes' shape in both scenarios (i.e., either with the magnitude of the penetration of the cylinder into the cube or with the amount of shape change across the cube's motion; also see Fakhourny, Culmer, & Henson, 2015). Then, Paulun et al. (2017) obtained stiffness ratings for the cubes in the first scenario (cylinder pushing into cube) and in the second scenario (cube set into motion) when rendered with different optical materials, ranging from hard (e.g., steel, copper) to soft (e.g., latex, velvet). They showed that although the optical appearance had a strong effect on the stiffness ratings of static images of the nondeformed cubes (i.e., a steel cube was judged much harder compared with a velvet cube), there was no systematic effect of the optical cues once the cubes were seen deforming. 

Overall, previous studies showed a reliable effect of shape cues on perceived stiffness when observers viewed simple objects subjected to a small range of shape transformations (i.e., free falling, indented, or wobbling objects). The apparent identity and similarity of deformable materials were strongly affected by their optical properties (Aliaga et al., 2015; van Assen & Fleming, 2016). However, the few studies that have also tested the effect of material appearance on perceived stiffness generally showed only weak effects of optical material when motion and deformation cues were present. 

Our study is motivated by the observation that optical cues should be a major factor in visual stiffness estimations. In Figure 1, we show two objects made out of hard materials (porcelain, wood) in which artists re-create the hallmark shape features of soft objects with striking verisimilitude (cf. Ludden, Schifferstein, & Hekkert, 2008). Still, as long as you know from which material the objects are made, you would assume that they would feel hard to your touch. In other words, the artists depict softness, but we do not literally perceive the materials to be soft, because of the appearance being overridden by semantic knowledge about the materials. If the object moved in way consistent with the behavior of a soft body (e.g., the bat would sink deeper into the material of the vase), this might increase the saliency and reliability of the soft shape features and thus change the judgment about the material and its stiffness. However, even then, it is hard to conceive that the visually perceived material identity will play no role at all for visual stiffness estimations. These are the types of intuitive judgments we want to study here. 

Figure 1

 

Examples in which hard materials (porcelain, wood) were used to create objects with shapes that appear soft. Note how we would not expect these static objects to feel soft as long as we know that they are made from porcelain and wood. “Iconocraste au bat I” by Laurent Craste (2010) and “Shroud” by Daniel Webb (2008). Reprinted with permission.

Specifically, we want to investigate the interplay between both types of visual inference about stiffness—association versus estimation—by contrasting optical material cues with shape deformation cues in static and dynamic scenes with unfamiliar objects. These objects were subjected to 12 different smooth transformations (“smooth” meaning that objects do not break into pieces, crumble, or splinter; cf. Schmid & Doerschner, 2017) and rendered with 28 different optical materials. We obtained stiffness judgments on static objects with varying materials and constant shape (Experiment 1), with varying shape and constant material (Experiment 2), with varying material and shape (Experiment 3), and finally obtained stiffness judgments on animated object transformations with varying material and shape (Experiment 4). 

Experiment 1: Material ratings

Experiment 1 was conducted to test the influence of optical material cues on perceived stiffness (i.e., the associative approach). For this purpose, we varied only the optical material but kept the shapes constant. Results of this experiment were also used to choose an appropriate stimulus range for Experiments 2, 3, and 4. 

Materials and methods

Participants

Twenty students from the Justus-Liebig-University Giessen, Germany, with normal or corrected vision participated in the experiment for financial compensation. All participants gave informed consent, were debriefed after the experiment, and treated according to the ethical guidelines of the American Psychological Association. All testing procedures were approved by the ethics board at Justus-Liebig-University Giessen and were carried out in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). 

Stimuli

Base objects:

Using Blender 2.76 (Stichting Blender Foundation, Amsterdam, the Netherlands), an open-source three-dimensional (3D) computer graphics application, we created four irregular base objects (Figure 2) with maximal vertical extensions of (a) 27.33 cm, (b) 23.76 cm, (c) 27.70 cm, and (d) 27.55 cm and maximal horizontal extensions of (a) 26.74 cm, (b) 32.68 cm, (c) 26.74 cm, and (d) 28.67 cm. 

Figure 2

 

The four base objects rendered with plastic material (which was rated in Experiment 1 as neither particularly soft nor hard).

Renderings:

The render engine used to generate the final images was Maxwell 3.0.1.3 (NextLimit Technologies, Madrid, Spain). The objects were positioned on a 5,700- × 5,700-cm ground plane, and the camera viewed them from the front and slightly from above. We used 28 different optical materials with diverse appearances modeled by eye (rather than physical measurements). They were selected to represent a wide range of hard and soft as well as common and uncommon materials and varied in their texture, reflectance, and translucency. Specifically, they were designed to approximate the following materials: black marble, white marble, porcelain, nickel, concrete paving, cement, ceramic, steel, copper, light wood, dark wood, silvered glass, glass, stone, leather, wax, gelatine, cardboard, plastic, paper, latex, cork, ice cream, lichen, waffle, denim, moss, and velvet. Some of these materials were downloaded or based on downloads from the Maxwell free resources library (http://resources.maxwellrender.com), and others were designed by us. The ground plane was rendered with a textured gray surface. The images were rendered at a resolution of 900 × 600 pixels and a sampling level of 18, and scenarios were lighted by a studio-like environment map. A selection of stimuli rendered for Experiment 1 is depicted in Figure 3. 

Figure 3

 

Stimuli for Experiment 1. The first base object rendered with each of the materials. Materials are ordered in their rated appearance in Experiment 1 from hard (upper left) to soft (lower right). For Experiment 1, each base object was rendered with each of the materials.

For Experiment 1, we rendered all four base objects with all optical materials, obtaining a total of 112 objects: 28 materials × 4 base objects. In all of the experiments, the height and width of each stimulus on screen were 15.5 × 26.0 cm (about 18° × 29° of visual angle). All stimuli are available for download at https://doi.org/10.5281/zenodo.290633. 

Procedure

Stimuli were presented on a black background on a Dell U2412M monitor at a resolution of 1,920 × 1,200 pixels, controlled by Matlab using the Psychophysics Toolbox extension (Kleiner, Brainard, & Pelli, 2007). The distance to the monitor was about 50 cm. 

Before the start of the experiment, participants were shown printouts with examples of base objects rendered with all 28 materials to provide them with information about the range of possible stimuli. In each experimental trial, participants were presented with a single object on the right of the screen and completed (a) a free material-naming task (“enter the name of the material”) and (b) a continuous rating task of 11 material attributes by adjusting a rating bar on the screen (soft, fragile, heavy, massive, realistic, large, crumbly, slippery, elastic, sticky, bendable; “rate the material on each scale”). See Supplementary Table S1 for the exact labeling of the scales and the definitions of the attributes as given in the instructions. Each participant named and rated 52 objects in random order, defined by just two of the base objects rendered with all 28 materials (i.e., not each combination of base object and material was rated by each participant). The chosen base objects and their combinations were counterbalanced across participants. 

Analysis

For the results of the rating task, we calculated a linear mixed model for the softness ratings with fixed effects of material, base object, and material × base object (necessary as not each combination of base object and material was rated by each participant). As a measure of interindividual differences, we calculated an interrater reliability score defined by the average correlation coefficient between softness ratings for each possible pair of participants. Also, we performed a principal component analysis including the 11 material attributes. 

The results of the free material-naming task were cleaned up by correcting spelling errors, by eliminating adjectives and reducing compound words to the noun (e.g., “crinkled cardboard” → “cardboard”; “iron plate” → “iron”). 

Results and discussion

Figure 4A shows the results of all ratings, with softness highlighted as this is the characteristic that is most important for the other experiments (Supplementary Figure S1 for average ratings for each material). We found large differences in perceived softness between the different optical materials: Black marble was, for instance, rated substantially harder than velvet, which was rated the softest. This was confirmed by a significant fixed effect for material, F(27, 1.099) = 41.83, p < 0.001, but not for base object or their interaction. The interrater reliability score for softness was r = 0.48. 

Figure 4

 

Results of Experiment 1. (A) For each rating scale, we show the three lowest-scoring, two medium-scoring, and three highest-scoring materials (exemplified by renderings of the first base object); black bars show mean ratings per material with [0, 1] set to the height of the panels. (B) Materials (dark gray) and material attributes (green; softness: blue) plotted onto the first two dimensions of a principal component analysis based on their loadings on these dimensions. The first two dimensions explain 73.45% of the variance, three dimensions explain 87.48%, and four explain 93.61% of the variance.

The other ratings were obtained to confirm that our selected materials span a wide range with respect to their material appearances and attributes: Indeed, we found that materials varied considerably on these attributes (Figure 4B). This shows that the range of optical materials used here was associated with different physical properties that are not primarily visual, such as weight or fragility. Thus, observers seem to be able to infer many material properties through an associative approach by assigning known attributes to visually classified material classes. 

In a control experiment in which another set of 12 participants rated hemispheres rendered with all 28 materials, we obtained similarly strong effects of material on softness ratings, F(27, 297) = 15.95, p < 0.001. Overall, the pattern of results was similar to that obtained with the four base objects, as indicated by the high correlation between mean material ratings of both stimulus sets, r(26) = 0.93, p < 0.001. Thus, our results are not specific to the type of objects we used in this study. 

Finally, Supplementary Figure S2 summarizes the results of the free material-naming task. Generally, they show that most of our renderings yielded compelling impressions of realistic materials that observers were able to reliably classify. In the following, the naming results were used to decide on the materials to test in Experiments 3 and 4 by choosing only materials that were identified as the same material by at least 50% of participants (see Stimuli section of Experiment 3). 

Raw data from all experiments are available for download at https://doi.org/10.5281/zenodo.290633. 

Experiment 2: Transformation ratings

The second experiment was conducted to test the influence of shape deformation cues on stiffness perception (i.e., the estimation approach). For this purpose, we varied the deformation of the four base shapes from Experiment 1 but kept the optical material constant. Because the material is constant and the base shapes are unknown, observers have to rely on estimation (rather than on association) to infer stiffness. 

Materials and methods

Participants

Participants were 24 students from the Justus-Liebig-University Giessen, Germany; other details were the same as in Experiment 1. 

Stimuli

Simulation:

For Experiment 2, we used all base objects rendered with the blue “plastic” material (which was rated as neither particularly soft nor hard in Experiment 1; M = 0.52, SD = 0.27, within the range [0, 1]). Each of the four base objects was subjected to different external forces to produce 12 different deformations, using RealFlow 2014 8.1.2.0192 (NextLimit Technologies, Madrid, Spain), a 3D simulation software that features a rigid and soft body dynamics engine (Caronte) for simulating deformable objects. In one scenario, this external force was gravity; in all other scenarios, the external force was applied by one or several other objects affecting the base object. As we are interested only in the states of the deformed base object and the perception of stiffness from its shape, we did not render these other objects (which would have provided additional cues for the identification of the type and magnitude of transformation). 

Generally, the objects were rigidly attached to the 5,700- × 5,700-cm ground plane and were simulated as soft bodies with a resolution of 125. The behavior of soft bodies in RealFlow is determined by several parameters, only some of which have equivalents in real-world physics. Most of the parameters were held constant across all simulations: The mass was 1.0 kg and elasticity was 0.0 (on a scale from 0.0 to 1.0, describing the amount of energy that is kept by the body on collision, i.e., the magnitude of bounces when it collides); internal damping was set to 6.0, which virtually eliminates bouncing; plasticity was turned on (i.e., the object was permanently deformed and would not recover its original shape); autocollision was turned off (i.e., the ability of different parts of the soft body to collide, which was not required for the shape and range of deformations we used); friction was set to 0.3 (on a scale from 0.0 to 1.0); and air friction was set to 0.005 (on a scale from 0.0 to infinity). For some of the scenes, some initial velocity or rotation was given to obtain the desired effect (e.g., to let the objects in the gravity scenario fall to the left). Length stiffness and volume stiffness are the recovery constants relative to the object (on a scale from 0.0 to 1,000.0) and determine the resistance of the object against changes in its original volume or its longitudinal magnitudes, respectively. In our simulations, we varied both values between scenarios to obtain the desired effects in the range between [0.002, 20] and [0.002, 20]. Note, however, that in a given scenario, both values were always the same (except scenario “I”) and constant for all base objects (Supplementary Table S2). In total, 52 objects were used in Experiment 2: the four base objects plus 12 transformations × 4 base objects. The viewpoint of the camera was the same as in Experiment 1. A selection of stimuli rendered for Experiment 2 is depicted in Figure 5. All stimuli are available for download at https://doi.org/10.5281/zenodo.290633. 

Figure 5

 

Stimuli for Experiment 2. Base object 1 subjected to each transformation A–L. Transformations are ordered in their rated appearance in Experiment 2 from hard (upper left) to soft (lower right). For Experiment 2, each of the base objects was subjected to each transformation; base objects are not shown here but were also included.

Procedure

Before the start of the experiment, participants were shown printouts with examples of objects subjected to the different transformations to provide them with information about the range of possible stimuli. In each experimental trial, participants were presented with a single object in the center of the screen. Half of the participants completed a softness rating task (“rate how soft the object appears to you”), with softness defined in the instructions as “the extent to which the object can be pushed in.” Again, note that this definition of softness, which was also used in Experiment 1, defines it in terms of perceived stiffness. The other half of the participants completed a deformation rating task (“rate how deformed the object appears to you”), with deformation defined in the instructions as “the extent to which the object has been deformed from its original state” (note that participants did not know the base object). Each participant rated all 52 objects in random order. Other details were the same as in Experiment 1. 

Analysis

We calculated a repeated-measures analysis of variance (rmANOVA) with the factors transformation and base object separately for the softness and deformation ratings. 

We also defined an objective measure of deformation by calculating the average Euclidean distance between object meshes. Specifically, for each of the 52 objects, we calculated the average distance of all vertex positions that were visible to the participants in the transformed object to the positions of the same vertices in the base object. Then, we built a grand average across all vertices to obtain a single deformation value for each object. We normalized these values across all objects to the range [0, 1] and correlated them with the average softness and deformation ratings (normalized to the range [0, 1] within each participant and then averaged). Note that this measure has limitations as it is based on pure Euclidean distance without taking vertex movement trajectory during the transformation into account. Nevertheless, we find that as a simple first-order measure, it captures the most important features of the deformations (Paulun et al., 2017). 

Results and discussion

Mean ratings and mesh deformation values for all transformations are plotted in Figure 6A (for results per base object, see Supplementary Figures S3–S6). For the softness ratings, we observed an effect of transformation, F(12, 132) = 3.11, p = 0.001, and base object, F(3, 33) = 5.00, p = 0.006, with no interaction effect. In other words, when optical material properties were held constant, the softness ratings were affected not only by the type of transformation but also by the base object (i.e., across all transformations, Objects 1 and 4 were rated softer compared with the two other objects). Note that mean ratings for the untransformed base objects were closer to the lower end of rating spectrum (no significant difference to the mean of the lowest-rated transformation), T(47) = 1.61, p = 0.113, compared with the higher end, T(47) = −4.71, p < 0.001. Thus, the base objects were on average perceived as rather hard compared with the other objects. The interrater reliability score for softness was r = 0.09, indicating large interindividual differences. 

Figure 6

 

Results of Experiment 2. (A, B) For each rating scale (softness, deformation), we show the transformations from scoring lowest to scoring highest (exemplified by renderings of case object 1). Black bars show mean ratings per transformation across base objects with [0, 1] set to the height of the panels. The violet frame marks the base object. (C) The mesh deformation is color coded between 0% (dark blue) to 100% (bright yellow) deformation. Black bars show the average mesh deformation per transformation across base objects with [0, 1] set to the height of the panels. The violet frame marks the base object. (D) Scatter plot for the softness ratings versus the deformation ratings. Green dots show the correlation between mean ratings for each object (normalized for each participant and then averaged); violet dots mark the ratings of the four base objects; the line is a least-squares fit to these data. (E) Scatter plot for the softness ratings versus the mesh deformation values. Same legend as in D. (F) Scatter plot for the deformation ratings versus the mesh deformation values. Same legend as in D.

For the deformation ratings, we found effects of transformation, F(12, 132) = 10.20, p < 0.001; object, F(3, 33) = 13.48, p < 0.001; and their interaction with the base object, F(36, 396) = 1.89, p = 0.002, implying that some transformations had stronger/weaker effects on the deformation ratings depending on the object they affected. Across all transformations, Objects 1 and 4 were rated more deformed compared with the two other objects. Note that mean ratings for the untransformed base objects were closer to the lower end of the rating spectrum (no significant difference to the mean of the lowest-rated transformation), T(47) = 0.29, p = 0.772, compared with the higher end, T(47) = −9.49, p < 0.001. Thus, the base objects were perceived as being deformed only little (in comparison with their transformed versions), suggesting that participants can detect the presence of telltale signatures of deformation. The interrater reliability score for deformation was r = 0.31. 

Finally, we found significant correlations between the softness ratings and the mesh deformation values, r(50) = 0.46, p < 0.001 (Figure 6E), as well as between the deformation ratings and the mesh deformation values, r(50) = 0.33, p = 0.017 (Figure 6F). Surprisingly, there is no significant correlation between the softness and deformation ratings, r(50) = −0.23, p = 0.094 (Figure 6D). This shows that participants' judgments about stiffness are affected by the physical deformation of the transformed objects relative to the base objects and that this deformation is to some extent accessible to participants. However, perceived softness and perceived deformation are not associated with each other. As indicated by the average ratings and interrater reliability scores, the effects of transformation on softness are quite weak and inconsistent; however, to the extent that it does vary, it correlates weakly with the degree of physical but not perceived deformation. We discuss this finding together with those of Experiments 3 and 4 in the final discussion section. 

Experiment 3: Material × transformation ratings (static images)

The purpose of Experiment 3 was to test the combined influence of optical material cues and shape deformation cues and to directly investigate the interplay between both types of visual inference about stiffness (association and estimation). This was done by combining a subset of the optical materials from Experiment 1 with a subset of the transformations from Experiment 2. 

Materials and methods

Participants

Participants were a new group of 12 students from the Justus-Liebig-University Giessen, Germany; other details were the same as in Experiment 1. 

Stimuli

We chose a subset of seven transformations and seven materials that should span a wide range of softness ratings (three × hard, one × medium, three × soft with respect to the rank order). We selected materials that had been identified as the same or similar material by at least 50% of participants in the free naming task of Experiment 1 and that were not too similar to any of the other six materials. Because of this latter criterion, (a) we chose only one type of marble and (b) we did not choose moss, denim, and waffle, as they were too similar to velvet in terms of the material being perceived as restricted to the object surface (i.e., rated as rather hollow compared with massive in Experiment 1). The seven transformations were A–C, F, and J–L, and the seven materials were black marble, porcelain, nickel, plastic, latex, cork, and velvet, producing a total of 196 objects: seven transformations × seven materials × four base objects. A selection of stimuli rendered for Experiment 3 is depicted in Figure 7. All stimuli are available for download at https://doi.org/10.5281/zenodo.290633. 

Figure 7

 

Example stimuli from Experiment 3. The first base object subjected to each transformation and rendered with each material. Materials and transformations are ordered in their rated appearance in Experiments 1 and 2 from hard (upper left) to soft (lower right). For Experiment 3, each of the base objects was subjected to each of the seven transformations and rendered with each of the seven materials.

Procedure

Before the start of the experiment, participants were shown printouts with examples of objects rendered with all materials and subjected to all transformations. The experimental procedure was the same as that of the softness rating task in Experiment 2. Each participant rated all 196 objects in random order. Other details were the same as in Experiments 1 and 2. 

Analysis

We calculated a rmANOVA with the factors transformation, material, and base object for the softness ratings. 

Results and discussion

We found a significant main effect of material, F(6, 66) = 30.87, p < 0.001, but no other effects (Figure 8), implying that stiffness perception was determined by the optical material cues without any role of the type of transformation (or the base object). Although the order of the materials was slightly different from that obtained in Experiment 1 (compare Figures 7 and 8), the correlation between the average softness ratings for materials in both experiments was high, r(5) = 0.96 and r(26) = 0.93, p < 0.001. The interrater reliability score for softness was r = 0.57. 

Figure 8

 

Results of Experiment 3. (A) For softness ratings, we show materials (first row) and transformations (second row; each exemplified by renderings of the first base object) from scoring lowest to scoring highest; black bars show mean ratings per material (across transformation) and transformation (across material) with [0, 1] set to the height the panels.

Experiment 4: Material × transformation ratings (animations)

Because previous research (e.g., Han & Keyser, 2015, 2016; Paulun et al., 2017) suggests a strong influence of shape deformation cues compared with optical cues when using moving stimuli, we further investigated the interplay between both types of cues with animations of the deformation. In Experiment 4, we again used a subset of transformations and optical materials, but this time, participants observed the deformation process (although not the deforming effectors or forces), which might make the shape cues more salient and reliable and also provides participants with knowledge about the shape of the untransformed object. 

Materials and methods

Participants

Participants were a new group of 13 students from the Justus-Liebig-University Giessen, Germany; other details were the same as in Experiment 1. 

Stimuli

We chose the seven transformations (A–C, F, J–L) and a subset of five of seven materials from Experiment 3 (porcelain, nickel, plastic, latex, velvet; for black marble and cork, dynamic renderings were not possible because of texture-mapping artifacts). For each transformation, we rendered the simulation as animations showing how the object transformed from the base object to the final state over 11 frames with a frame rate of 30 fps, obtaining a total of 140 animations: seven transformations × five materials × four base objects. 

Procedure

Participants rated (a) static images of the base objects rendered with all five materials followed by (b) dynamic animations of the seven transformations unfolding over time rendered with all five materials in two different sections of the experiment. Before each of the two sections, participants were presented with examples of stimuli from that section. Each animation was presented in a loop, starting with a short presentation of a random noise mask (167 ms), followed by 30 repetitions of the first frame (static base object; 1000 ms), the 11 frames of the animation (367 ms, with 33 ms each), and concluding by 10 repetitions of the last frame (static transformed object; 334 ms; see Figure 9). This gave a compelling impression of the movement process as a whole and prevented apparent motion between the final state and the initial state at the loop point. Each participant rated all 20 static objects (five materials × four base objects) in random order, followed by all 140 animations in random order (for examples, see Supplementary Movies S1–S5. Other details were the same as in Experiments 2 and 3. All stimuli are available for download at https://doi.org/10.5281/zenodo.290633. 

Figure 9

Example of image sequence in one trial of Experiment 4. The sequence was repeated in a loop until participants rated the softness of the object (using a rating scale presented just below the images). For a movie file of this example and examples of the other four materials for different transformations and base objects, see Supplementary Movies S1–S5. 

Analysis

We calculated a rmANOVA with the factors material and (base) object for the softness ratings of the static images and one including the additional factor transformation for the softness ratings of the animations. 

For the animations, we again calculated the average Euclidean distance between object meshes of untransformed and transformed objects as an objective measure of deformation (see Experiment 2). For each object, we built a grand average across all vertices and frames, normalized these values across all objects to the range [0, 1], and correlated them with the average softness ratings (normalized to the range [0, 1] within each participant and then averaged, across all materials). 

Results and discussion

For the static images, we observed a main effect of material, F(4, 48) = 35.94, p < 0.001, and no effect of base object (Figure 10A; interrater reliability score, r = 0.69). This replicates our results from Experiment 3. 

Figure 10

 

Results of Experiment 4. (A) For softness ratings of static images, we show materials (exemplified by renderings of the first base object) from scoring lowest to scoring highest; black bars show mean ratings per material (across objects) with [0, 1] set to the height of the panels. (B) Softness ratings of animations; for details, see (A). (C) Softness ratings of animations as a function of transformation per base object (different colors).

For the animations, we observed main effects of material, F(4, 48) = 4.08, p = 0.006 (Figure 10B); transformation, F(6, 72) = 11.62, p < 0.001; and object, F(3, 36) = 6.71, p = 0.001. Note that the softness ratings per material, when compared with those for static images, were higher across the board, even for the hard materials nickel and porcelain. Thus, when directly observing the deformation of a shape, deformation cues influence the perceived softness of an object. This effect occurred in addition to the influence that the optical material had on perceived softness. Furthermore, similar to Experiment 2, there was also an effect of the shape of the base object, where across all transformations and materials, base Objects 1 and 3 were rated softer compared with the two other objects. Finally, the interaction of transformation and base object was significant, F(18, 216) = 11.36, p < 0.001 (Figure 10C), showing that the same transformations applied to different base objects produced different softness ratings. No other effects were significant. The interrater reliability score for softness was r = 0.21. 

Finally, we found no significant correlation between the softness ratings and the average mesh deformation values. Thus, in contrast to the findings in static images (Experiment 2), participants' judgments about stiffness were not affected by the physical deformation of the transformed objects relative to the base objects. This finding is inconsistent with earlier findings by Paulun et al. (2017), who did report a strong correlation between average physical deformation and softness ratings for animated transformation sequences. 

General discussion

To test the interplay between two types of visual inference about stiffness—association versus estimation—we subjected four unfamiliar base objects to a number of smooth transformations and rendered them with a wide range of optical materials. Then, we obtained stiffness judgments on static objects with varying materials and constant shape (Experiment 1), with varying shape deformations and constant material (Experiment 2), and with varying material and shape (Experiment 3), and we finally obtained stiffness judgments on animated object transformations with varying material and shape (Experiment 4). Consequently, we could test the relative influence of material compared with static shape cues on stiffness perception. By comparing static images and animated transformations, we could also test for the relative weight of material and shape cues in both situations. Based on previous findings, we should expect a weak role of material cues and a strong effect of physical deformation in the dynamic scenes. 

Optical, shape, and motion cues in stiffness perception

Compared with previous studies, we find a much less pronounced effect of shape cues compared with material cues (Han & Keyser, 2015, 2016; Paulun et al., 2017). In static objects, shape cues play a role only when the material of the objects is constant (Experiment 2). For static objects, if both deformation and material vary, observers no longer rely on shape cues to judge stiffness (Experiment 3). However, when the same transformations are animated in dynamic scenes, shape cues dominate estimations of stiffness (Experiment 4), in line with previous studies. How can these different findings be reconciled? 

We suggest that a reliability-weighted cue combination can explain the varying findings across our and previous findings. In line with the idea that the visual system weights the incoming information by its reliability (e.g., Hillis, Watt, Landy, & Banks, 2004; Jacobs, 1999), optical and shape cues are assigned different weights depending on the relative certainty with which they provide information about the stiffness of objects. 

In contrast to most previous studies, we used unfamiliar base objects (versus simple spheres or cubes) and invisible transformation effectors and also tested the role of shape deformation cues in static scenes. All of these factors make shape a less reliable indicator of deformation, for example, by precluding comparisons between pretransformation and posttransformation object shapes. Consequently, shape contributes little weight to the combined estimate of stiffness. Instead, the visual system relies on the information that can be identified with more certainty, namely, the optical material. 

In other scenarios—with familiar base shapes or animated transformations in dynamic scenes (e.g., Experiment 4; Paulun et al., 2017)—shape cues gain more weight. For example, animating the transformations strengthens the perceived changes in shape: a number of previous studies demonstrate a key role of motion cues in the estimation of material properties (e.g., Bi & Xiao, 2016; Bouman, Xiao, Battaglia, & Freeman, 2013; Kawabe et al., 2015; Kawabe & Nishida, 2016; Masuda et al., 2013; Masuda, Matsubara, Utsumi, & Wada, 2015). Consequently, in these scenarios, shape cues dominate estimations of stiffness. 

Thus, the relative weighting of optical and shape cues varies significantly across contexts. For example, we find that shape cues in static objects are effective only when optical cues are constant (Experiment 2) but not when optical cues vary at the same time as shape (Experiment 3). In the dynamic scenes of Paulun et al. (2017), shape cues are so strong that they dominate optical cues even without visible transformation effectors (i.e., when cubes were set into reverberating motion). In contrast, Aliaga et al. (2015) asked participants to compare hybrid cloth stimuli (e.g., with the appearance of cotton but the dynamics of silk) with veridical animations of cloth (e.g., with the appearance and dynamics of cotton) and found that matches for most stimuli depended on optical rather than dynamic shape cues. However, for stimuli with very characteristic motion dynamics (i.e., silk), shape cues were more important, presumably because the shape changes are so distinctive that they are given more weight by the visual system. 

This implies that we should be careful in generalizing findings from single studies on the role of optical and shape cues in the estimation of material properties. For example, our experiments were limited by the fact that we used a quite homogeneous class of smooth transformations (i.e., objects did not break into pieces, crumble, or splinter; cf. Schmid & Doerschner, 2017). For transformations that are characteristic of a specific class of objects/materials (e.g., only hard, fibrous materials like wood tend to splinter), effects might have been stronger and interindividual differences smaller. In line with this, van Assen and Fleming (2016) found no effects of the optical appearance of fluids on ratings of viscosity—another mechanical property—in either static or dynamic scenes. This probably results from the fact that the shape of fluids is very characteristic; hardly any solid objects are shaped like liquids. 

The role of perceived deformation in stiffness perception

Previous work using animations of deforming cubes reports a strong correlation between average physical deformation and perceived stiffness (Paulun et al., 2017). Here, in contrast, we find a much less pronounced effect of deformation on perceived stiffness. We find no correlation between judgments of deformation and stiffness in static images (Experiment 2), suggesting that inferences about deformation and stiffness were not based on the same stimulus features. Moreover, we find only a rather weak correlation between the ground truth deformation of our stimuli and their perceived stiffness in static images (Experiment 2) and no correlation at all in dynamic scenes (Experiment 4). How can the difference between our findings and previous studies be explained? 

When an object is shown deforming (as in Paulun et al., 2017), the magnitude of deformation can be inferred by comparing the final state of the object with its initial state (Schmidt & Fleming, 2016). In contrast (as in the current study), determining that an unfamiliar, static shape has been deformed from some unseen original shape is nontrivial. Despite this, the correlation between deformation and perceived softness indicates that participants can detect certain telltale signatures of deformation in the shapes. This is supported by relatively low softness and deformation ratings for base objects, suggesting that participants establish some internal prototype from generic assumptions about the shape of untransformed objects (e.g., symmetric, within a typical range of width-to-height ratios, parts of similar length and thickness) and from adjustments to the stimulus set at hand. 

On the other hand, that the correlation is low shows that the way we calculated physical deformation might not be optimal for our range of stimuli. For example, the mean deformation of Transformation B of base Object 1 (Figure 11A) seems to reflect well the perceived transformation, that is, a stretch of the object's “head.” However, in Transformation C of base Object 1 (Figure 11B), the largest calculated average deformation is at the tip of the left limb, because its vertices are farthest away from their original positions in Euclidian terms. However, observers might perceive the concavity on the object's right side as the area of the largest deformation because this part has changed its shape and not just its position (from straight to curved). Thus, in our stimuli, the area and magnitude of perceived deformation might be different from the area and magnitude of physical mesh deformation based on Euclidean distance. 

Figure 11

 

Illustration of potential differences between perceived and mesh deformations. (A) Object in which areas of maximal mesh deformation (lighter shades of blue) correspond to areas of maximal perceived deformation (red lines; arrows indicate transformation direction). (B) Object in which the area of maximal mesh deformation is different from the area of maximal perceived deformation.

Finally, the null correlation between stiffness and deformation ratings show that under circumstances in which participants are uncertain about the magnitude of deformation, their estimations of stiffness are not systematically related to perceived deformation, which probably results in the large interindividual differences (see stiffness ratings of Experiment 2). 

Limitations

Our scenes were simulated without gravity, which might also have affected participants' responses. For example, when taking gravity into account, the range of stiffness ratings might be reduced: Participants might reason that all of the objects must have some basic level of stiffness (because they are not collapsing under their own weight, their parts are not sagging, etc.). Nevertheless, it should be noted that our primary aim was not to mimic natural behavior of soft objects but to produce scenes with strong shape cues; simulations without gravity made it easier to produce malleable objects that did not behave like deflating balloons. 

Although we instructed our participants to rate the softness of objects by defining it effectively as stiffness (i.e., the “extent to which the object can be pushed in”), softness is not a well-defined term in everyday use. Indeed, it might also refer to a surface property (e.g., although an object might be very stiff, it still feels soft to the touch when it is covered in velvet). This ambiguity of the term softness might have given optical material cues more weight in the obtained ratings. However, note that the wording of our instructions was very close to that used by Paulun et al. (2017), who reported no effects of optical material cues on softness ratings. 

Conclusion

In conclusion, our findings suggest that observers integrate shape, motion, and optical cues when inferring stiffness, effectively combining an associative and estimation approach. In contrast to previous studies, we show that optical material cues play a strong role even when judging stiffness in dynamic scenes, whereas the role of (perceived) deformation is rather small. The visual system has access to a set of different cues to stiffness and seems to adaptively vary its weighting and use depending on the presence and reliability of these cues in the visual input. 

Future research should investigate how the human visual system determines the reliabilities of these cues and should systematically identify their weights. By doing this, it could also be tested whether cue integration in stiffness estimation is consistent with the standard weak fusion (or Bayesian) model (Landy, Maloney, Johnston, & Young, 1995). Little is known about which shape features the visual system uses to judge deformable objects—further studies should investigate the cues we use to identify how and to what extent objects have been deformed from some unseen initial state. It is also interesting to ask which other factors contribute to the extraction of these “signature features” (such as shape regularity) and, finally, how those features are used to infer the material properties (such as stiffness), the original undistorted object shape, and potentially also the entire process that formed the new shape, from just a single snapshot. 

Acknowledgments

This research was funded by the DFG-funded Collaborative Research Center “Cardinal Mechanisms of Perception” (SFB-TRR 135) and the ERC Consolidator award “SHAPE” (ERC-CoG-2015-682859). We wish to thank Harun Karimpur and Eugen Prokott for data collection and Eugen Prokott for assistance in material design. 

Commercial relationships: none. 

Corresponding author: Filipp Schmidt. 

Email: . 

Address: General Psychology Otto-Behaghel-Str. 10F, 35394 Gießen, Germany.