Shape, motion, and optical cues to stiffness of elastic objects

Abstract

Nonrigid materials, such as jelly, rubber, or sponge move and deform in distinctive ways depending on their stiffness. Which cues do we use to infer stiffness? We simulated cubes of varying stiffness and optical appearance (e.g., wood, metal, wax, jelly) being subjected to two kinds of deformation: (a) a rigid cylinder pushing downwards into the cube to various extents (shape change, but little motion: shape dominant), (b) a rigid cylinder retracting rapidly from the cube (same initial shapes, differences in motion: motion dominant). Observers rated the apparent softness/hardness of the cubes. In the shape-dominant condition, ratings mainly depended on how deeply the rod penetrated the cube and were almost unaffected by the cube's intrinsic physical properties. In contrast, in the motion-dominant condition, ratings varied systematically with the cube's intrinsic stiffness, and were less influenced by the extent of the perturbation. We find that both results are well predicted by the absolute magnitude of deformation, suggesting that when asked to judge stiffness, observers resort to simple heuristics based on the amount of deformation. Softness ratings for static, unperturbed cubes varied substantially and systematically depending on the optical properties. However, when animated, the ratings were again dominated by the extent of the deformation, and the effect of optical appearance was negligible. Together, our results suggest that to estimate stiffness, the visual system strongly relies on measures of the extent to which an object changes shape in response to forces.

Introduction

Humans readily identify a wide variety of different materials based on visual information. Most previous research has focused on optical properties of materials—for example, what two materials must have in common to appear similarly glossy (Anderson & Kim, 2009; Kim & Anderson, 2010; Motoyoshi, Nishida, Sharan, & Adelson, 2007; Wiebel, Toscani, & Gegenfurtner, 2015) or translucent (Fleming & Bülthoff, 2005; Fleming, Jäkel, & Maloney, 2011; Motoyoshi, 2010). However, many other material properties—like stiffness, elasticity, or viscosity—are not indicated by how they interact with light, but rather with external forces (e.g., with gravity or other objects). Consider, for instance, an object lying still and in isolation versus one being prodded by another object. Although we may have certain expectations about the physical properties of the static objects—based on our previous experience with other objects made of materials with similar optical properties—only the latter scene can reveal whether the object is actually rigid or elastic. The challenge for the visual system is to identify image measurements that are diagnostic of the material (e.g., stiffness), while remaining invariant across features that are specific to the individual object (e.g., its original, undistorted shape), or external forces (e.g., momentum of the collision object). 

Shape is usually found to dominate object recognition (Logothetis & Sheinberg, 1996). Therefore, objects that routinely change shape potentially pose a challenge for the visual system. However, this deformation might actually be helpful in determining the material of the object, and potentially also provides information about the processes that have formed its present shape (Leyton, 1989; Pinna, 2010; Spröte & Fleming, 2015). Figure 1 illustrates this point: In Figure 1A it is easy to recognize the object as a ball because of its shape. Based on previous experience with balls, one might even expect it to be elastic, but the image itself tells only little about the material the ball is made of. Figure 1B, on the other hand, provides less information about the typical shape since the spherical form of the ball is distorted; however, this distortion, in turn, provides information about the material properties of the ball, as well as the forces to which it is being subjected. This estimate might not be exact (because of the ambiguity between external force of the grip and internal properties of the ball), but it is easy to see that the ball is neither fragile nor very stiff. 

Figure 1

 

Two different images of the same ball. In (A) the ball is easy to recognize as such because of its shape, but there is little information about its internal properties. In (B) the spherical shape is distorted, but the distortion reveals information about its stiffness.

In general, nonrigid materials like rubber or jelly respond to external forces by bending, bulging, and wobbling in distinctive ways. The specific manner a bouncy ball deforms on the ground or a piece of jelly wiggles on a spoon seem to provide rich information about the internal properties of the materials. Such characteristic deformations and motions could be used as visual cues to the stiffness of elastic objects. It has, for instance, been shown that small changes in the contour of simple two-dimensional (2-D) outlines (i.e., changes in shape) can change the perceived material properties and can make an object look softer (Pinna & Deiana, 2015). Even illusory contours can appear to belong to an elastic object when oscillating with the appropriate amplitude and frequency (Masuda, Matsubara, Utsumi, & Wada, 2015). Here we sought to investigate in greater detail the sources of information the visual system uses to determine the stiffness of deformable objects. 

Shape and motion have already been shown to provide important visual information about another mechanical material property—fluid viscosity; a liquid of a given viscosity has a certain flow pattern and settles into shapes with characteristic features. More specifically, Kawabe, Maruya, Fleming, and Nishida (2015) demonstrated that different viscosities can be identified from motion information alone (when no shape cues are available), and in a complementary study, Paulun, Kawabe, Nishida, and Fleming (2015) showed that simple measures of the shape of liquids predict their perceived viscosity very well in static snapshots (when no motion cues are available). Recent research, on the other hand, showed optical properties of liquids barely influence the perception of viscosity, especially when both motion and shape cues are available (van Assen & Fleming, 2016). The question we seek to investigate here is which cues are used to judge another important class of shape-changing materials, namely deformable solids, like rubber, jelly, or dough. 

Before reviewing the relevant literature on stiffness perception, it might be helpful to differentiate several related concepts: The stiffer (or more rigid) an object is, the smaller its deformation in response to a given force applied. If the object is elastic, the deformation is nonpermanent—that is, after the force has been removed the object will (partly) recover its original shape and the more elastic an object is, the closer its recovered shape will be to the original one. In contrast, materials that retain their deformed shape are plastic. The higher the elasticity of an object is, the lower its stiffness. However, low stiffness materials are not necessarily elastic (e.g., clay, which is plastic). The inverse of stiffness is called compliance; this term is most often used in the haptic literature. In the visual literature, the general term elasticity is often used and then mostly refers to the elastic modulus. Similar to stiffness, this is a measure for the resistance of elastic deformation in response to an applied force, but it refers to the material irrespective of the size or shape of the object. In an everyday understanding, such differentiation is hardly made, so it seems fair to compare judgments of elasticity to judgments of stiffness. The term softness is used here to describe the perceptual correlate of the inverse of stiffness. 

Most research about the perception of these material properties has been conducted in the haptic domain (see Di Luca & Ernst, 2014, for an overview). In the visual domain, most studies in this field have focused on judgments about the perceived elasticity of objects falling on a rigid ground. For example, Warren, Kim, and Husney (1987) reported initial evidence that observers use some simple heuristics when judging the elasticity of bouncing balls, such as the relative height of subsequent bounces. Nusseck, Fleming, Langarde, Bardy, and Bülthoff (2007) expanded these findings by showing that the choice of heuristics depends on whether one passively perceives a scene or has to predict the future behavior of the ball. Rules of thumb, like the height of a bounce, only apply to the specific scene of a bouncing sphere, of course—more complex shapes tend to rebound in less predictable ways, potentially making it harder to use such simple heuristics to infer elasticity. It is also worth noting that most previous investigations of elasticity perception have dealt with conditions in which deformations of object shape are either entirely absent or very small compared to the overall motion of the objects. In the current study, by contrast, we focused on shape deformations with objects that were rigidly attached to the ground plane, so that the rigid component of the motion was much smaller. Kawabe and Nishida (2016) also studied the perception of an elastic object falling on a rigid plane, but investigated image cues to elasticity other than those related to the bounces. More specifically, they investigated the contribution of two types of motion: motion from the deformation of the object's outlines and motion from the optical deformation within the translucent object. They found that observers can use both types of information to distinguish different levels of elasticity and the differences between the two cues may be related to differences in the optic flow pattern of both types of motion. This leaves the questions of (a) how observers judge stiffness in different types of scenes (not free falls); (b) how other optical material properties influence the judgment; and (c) more generally, what other visual cues to stiffness there are. These questions are addressed in the current study. Han and Keyser (2015, 2016) investigated the effects of high- and low-level texture information on the perception of deformation in another free-fall scene. They showed that high-level properties have little influence on the perception of deformation—for example, the deformation of a sphere is detected equally likely when the sphere has the optical appearance of a soccer ball or that of a billiard ball (Han & Keyser, 2015). In contrast, the detection of deformation may be facilitated through the low-level features of the texture, like contrast and spatial frequency (Han & Keyser, 2016). Scenes in which an elastic object falls down are in some sense less complex because the external force (gravity) is at least potentially known to the observer—as we are highly familiar with the acceleration of falling objects. Despite this, changes in gravity are not always detected in such free-fall scenes (Twardy & Bingham, 2002). This may reflect a depth-scaling ambiguity in the mapping from physical speed to retinal speed: A distant object falling from a great height will produce lower retinal accelerations than a nearby object that falls the same visual angle (a much smaller physical distance). Again, in the present study we explicitly wanted to investigate visual cues in other types of scenes. Fakhourny, Culmer, and Henson (2015) used a scene in which soft objects are indented by another object and found that the resulting deformations can be interpreted more accurately when all objects are indented with the same force rather than to the same amount. Thus, in this study the observers showed some bias in how they interpret the ambiguity between external force and internal material properties. In addition to investigating how accurately observers can estimate stiffness, in our study we also tried to understand how they derive this estimate. 

In sum, in this study we aimed to investigate specific visual cues to the stiffness of elastic objects, focusing on shape, motion, and optical appearance. Using computer graphics we generated a series of cubes with parametrically varying stiffness, and with carefully controlled interactions with other objects (specifically the cubes were poked in two different ways with a simulated rigid rod, while remaining rigidly attached to the ground). These different interactions emphasized two possible visual cues related to either shape or motion (Experiment 1). Since all stimuli were computer generated, we were also able to investigate the physical deformation patterns of all stimuli to identify cues that correlated with participants' softness judgments. In a second experiment we tested how objects, which were judged to be hard or soft in Experiment 1, were perceived when we varied their optical appearance (while holding the deformation constant across optical materials). We thus investigated how different optical appearances contribute to the visual perception of an object being softer or harder. 

Experiment 1: Shape and motion cues

Materials and methods

Stimuli

Two sets of stimuli were created using RealFlow 2014 (V.8.1.2.0192; Next Limit Technologies, Madrid, Spain), a dynamics simulation tool for three-dimensional (3-D) computer graphics. Each stimulus set showed one of two scenes. Both scenes consisted of a deformable cube, a rigid cylinder, and a ground plane, and were illuminated by a studio-like environment map. The difference between them was the way the rod interacted with the cube. The cube was 7 × 7 × 7 cm and was rendered with a blue, slightly translucent material, which gave it an appearance like silicone rubber. The cube was simulated as a soft body with a resolution of 125 (i.e., 125 voxels were used to bend the polygons of the mesh correctly). Several parameters determine the behavior and appearance of simulated soft bodies in RealFlow, only some of which have an equivalent in real-world physics or are measured in the same units. We kept most of the parameters constant across all simulations: The mass of the cube was set to be 0.343 kg (which resembles the density of silicone), its elasticity was set to 0.5 (on a scale from 0.0 to 1.0, describing the amount of energy that is kept by the body when it collides, in other words, the magnitude of bounces when it collides); internal damping (which is closely related to elasticity as both control the amount of internal motion, but which mainly influences the time after which the object stops bouncing as well as the size of the bounces) was set to 0.01, which allows bouncing for up to several seconds; plasticity was turned off (i.e., the cube could not be permanently deformed but would recover to its original shape); autocollision was also disabled (i.e., the ability of different parts of the soft body to collide, which was not required for the shape and range of deformations we used); friction was set to 0.3 (on a scale from 0.0 to 1.0); air friction was set to 0.005 (on a scale from 0.0 to infinity); and no initial velocity or rotation was given. Length stiffness and volume stiffness are the recovery constants relative to the object (on a scale from 0.0 to 1000.0), and determine the resistance of the object against changes in its original volume or its longitudinal magnitudes, respectively. In our simulations, we varied both stiffness parameters simultaneously in five steps: 0.01, 0.04, 0.16, 0.64, and 2.56. Because these values are (a) nondimensional, (b) have different effects depending on the number of edges in the mesh, and (c) have no direct analogue in physics, we will refer to different stiffness levels on an ordinal scale: 1, 2, 3, 4, and 5 (with 5 being the stiffest). Note, however, that despite the lack of a correspondence between simulation parameters and the parameters physicists use to represent elastic materials, the resulting simulations yield compelling impressions of realistic deformable materials with different physical properties. 

The cube was placed on a 1000- × 1000-cm plane with a glossy white surface. The camera viewed the scene diagonally from above one corner of the cube. The bottom side of the cube was attached at four points to the plane, so that it would stay at its initial position after collisions. The third object in the scene was a cylinder with a length of 8 cm and a diameter of 2 cm and a gray surface. Thus, the basic setup of both scenes was the same, but we chose two different types of interaction to emphasize two different possible visual cues to the softness of the cube that were either related to the shape deformation (shape cue scene) or to the motion of the cube (motion cue scene; see Figure 2). 

Figure 2

 

Example images of the experimental stimuli. (A) Shows the last frame of each of the 25 animations of the shape cue scene; the two highlighted example images are shown larger on the right side. Part of the background was cropped for this illustration so that the cubes could be depicted larger. (B) Shows an overlay of the second and third frame (out of 31 frames) of each animation of the motion cue scene to give an impression of the motion. Again, part of the background was cropped for this illustration so that the cubes could be depicted larger. The two images on the right side show different perspectives of the scene: a side view with the rendered cylinder, here the softest object and the largest perturbation depth (left); the same cube but from the camera perspective (i.e., the perspective of the participant; right). The cylinder was pushed into the cube for five frames (not shown to participants) and then released within one frame to set the cube in motion. The frame shown here is the first one of the animation that was presented to the observers. Movies of the stimuli are available in the supplementary material.

In the shape cue scene the cylinder was positioned 5 cm above the cube, with its long axis oriented horizontally, and moved downwards with a constant speed of 0.2 cm per frame (i.e., 0.06 m/s; see Figure 2A). We varied when this movement stopped (i.e., how deep the cylinder pushed into the cube) in five steps: 1.0, 1.5, 2.0, 2.5, or 3.0 cm. This is equivalent to varying the force with which the cylinder pushes into the cube. In this scene (although obviously motion was also present) the shape deformation of the cube was the dominant cue, and we wanted to test whether participants would use this information when judging the softness. Specifically, there was practically no reverberation at all: Each frame was simply a different stable state of the interaction between the rod and the cube. 

In the motion cue scene we set the cube in motion by releasing it from the deformation caused by the initial position of the cylinder, which was pushed into the rear edge that was not visible to the observer (we simulated, but did not render the cylinder; see Figure 2B). More specifically, we simulated the cylinder moving into the cube by different amounts within five frames: 2.7, 3.2, 3.7, 4.2, or 4.7 cm. After these five frames, the cylinder was immediately released (i.e., moved entirely out of the cube in one frame), resulting in a reverberating motion of the cube as it returned to its initial state. The animations we showed to our participants started at this release frame. Thus, in this scene the motion of the stimuli (i.e., the deformation of the objects over time) was the most dominant cue. 

All animations lasted 31 frames and were rendered with a frame rate of 30 fps. Each frame was rendered with the Maxwell renderer (V.3.0.1.3; Next Limit Technologies, Madrid, Spain) using a sampling level of 18, and saved as an 800- × 600-px .PNG image. In total we rendered 50 animations: 2 Cues (motion vs. shape) × 5 Stiffness Levels × 5 Depths. Stimuli were presented on a laptop (Lenovo IdeaPad Z570; screen resolution: 1366 × 768 px; refresh rate: 60 Hz), with a glossy LCD display at a freely chosen viewing distance (roughly 50 cm; i.e., the cube subtended roughly 8.5° of visual angle). All stimuli are available for download at https://doi.org/10.5281/zenodo.155310. 

Participants

Twenty-four observers participated in this study, 12 male and 12 female. They were on average 27.4 ± 5.2 years old (M ± SD). Twelve of the participants were presented with the motion cue scene (four female, eight male); the other 12 participants with the shape cue scene (eight female, four male). All participants were naive with regard to the aims of the study and gave written informed consent prior to participation. The experiment was conducted in accordance with the Declaration of Helsinki, and the procedure was approved by the local ethics committee LEK FB06 at Gießen University. 

Procedure

Prior to the experiment participants were told that they would have to judge the softness of objects presented on the screen. They were given two real cylindrical blocks of blue silicone with different compliances, to touch and get an impression which characteristic of the material they were subsequently supposed to rate visually. Observers were then presented with the four most extreme animations on the screen to show them the whole range of stimuli. On each trial, participants had to press a button to start the animation. After each animation a horizontal rating bar appeared under the image of the last frame, which stayed on the screen until the rating was done. Participants could adjust the slider to where they perceived the cube to be on a scale from soft to hard. They confirmed their adjustment with a mouse click and continued to the next trial. All 25 animations were presented three times in random order, resulting in 75 trials per participant. The experiment took approximately 5 min. 

Results and discussion

The rating data was on a scale between 0 (hard) and 1 (soft), depending on where the observers had set the slider during the experiment. For both scenes, data from each participant was averaged across the three repetitions to gain one mean rating per condition from each observer. Raw data from all experiments can be downloaded here: https://doi.org/10.5281/zenodo.155310. 

Shape-dominant condition

The average rating for each cube is shown in Figure 3A, with light shades of blue indicating harder ratings and darker, more saturated blues indicating softer ratings. Somewhat counterintuitively, the cube's apparent softness was almost entirely determined by how deeply the cylinder was pushed into it, rather than the simulated stiffness of the cube, as apparent from the horizontally striped pattern of results. This observation was confirmed by a 5 (stiffness) × 5 (depths) repeated-measures analysis of variance (ANOVA), which revealed a significant main effect of perturbation depth, F(1.14, 12.51) = 21.38, p < 0.001, η² = 0.442 (see Figure 3B), but neither a main effect of stiffness, F(2.66, 29.22) = 1.56, p = 0.223, η² = 0.003 (see Figure 3C), nor a significant interaction between the two factors, F(4.25, 46.74) = 0.76, p = 0.562, η² = 0.009. All reported degrees of freedom in this study were Greenhouse–Geisser corrected, because we could not always test whether the sphericity assumption was violated (as the number of conditions in some cases exceeded the number of subjects). Effect sizes are reported as eta squared (η²) and thus also rather conservative (Levine & Hullett, 2002). According to Cohen's guidelines (Cohen, 1988) they can be interpreted as a small (η² ≥ 0.01), medium (η² ≥ 0.059), or large (η² ≥ 0.138) effects. The effect of perturbation depth is thus very large and accounts for 44.2% of the variance in the data. With a perturbation depth of 3.0 cm, the cube was on average rated very soft with a value of 0.64 ± 0.06 (M ± 1 SEM), compared to 0.49 ± 0.05 at a depth of 2.5 cm, 0.39 ± 0.5 at a depth of 2.0 cm, 0.27 ± 0.06 at a depth of 1.5 cm, and 0.15 ± 0.7, thus rather hard, at a depth of 1.0 cm. Post hoc pair-wise comparisons showed that all depth levels were significantly different from one another, except 2.0 and 2.5 cm (all other ps < 0.05). It is important to note, however, that the chosen stiffness values of the simulation are not perceptually indistinguishable per se (see motion cue scene). The observed pattern of responses—with the striking insensitivity of observers to the actual stiffness of the material—is specific to this scene. 

Figure 3

 

Results of the perceptual rating of the cube's softness in the shape cue condition. (A) Shows the mean ratings (averaged across participants) as a function of simulated stiffness of the cube and perturbation depth of the cylinder color-coded from hard (white) to soft (dark blue). (B) Shows the mean ratings as a function of stiffness split up for different depth levels. (C) Shows the mean ratings as a function of perturbation depth split up for different stiffness levels.

These findings have several implications. First, the amount of deformation of an object seems to be an important and vivid visual cue to softness, irrespective of how soft the object really is. Second, the same cube can appear to be made of a different material that is soft or hard, depending on whether it is more or less deformed (at least in the absence of any additional information about how much force was applied to the material). Third, equivalently, the participants may have assumed the same constant force moving the cylinder in all conditions, thereby interpreting the penetration depth to be entirely due to the extent of resistance provided by the material. Thus, all participants resolved the inherent ambiguity between the external force and the internal properties of the cube in the same manner, by attributing all visual differences between the cubes to differences in its material, rather than the forces behind the cylinder. However, it should also be noted that participants were only asked about the properties of the cube, not the cylinder, so there is no explicit measure of this assumption, nor did we direct their attention specifically towards the cylinder. 

Motion-dominant condition

The pattern of results looked substantially different for the motion cue scene (see Figure 4A). Here, the simulated stiffness had a systematic effect on the perceived softness (significant main effect of stiffness: F[1.59, 17.52] = 90.58, p < 0.001, η² = 0.784; see Figure 4C), while the effect of perturbation depth was much weaker. The stiffer the material was, the harder it was perceived: The stiffest cube was rated the hardest with a mean value of 0.12 ± 0.03 (M ± 1 SEM), compared to the second stiffest material with a value of 0.29 ± 0.06, the Stiffness Level 3 with a value of 0.44 ± 0.05, the Stiffness Level 4 with a value of 0.70 ± 0.03, and the lowest stiffness level with a rating of 0.82 ± 0.03. Post hoc pair-wise comparisons showed that the differences between all levels of stiffness were significant (all ps < 0.05, Bonferroni-corrected). In addition to the effect of stiffness, there was also a significant main effect of perturbation depth on the perceived softness, F(2.37, 26.06) = 15.03, p < 0.001, η² = 0.172 (see Figure 4B). The deeper the cylinder was pushed into the cube (i.e., the larger the initial displacement of the cube) the softer it appeared, although this effect was much weaker than in the shape cue scene (as indicated by the smaller effect size). Paired comparisons showed a significant difference between a depth of 2.7 cm (0.40 ± 0.03) compared to 3.7 cm (0.48 ± 0.03), 4.2 cm (0.49 ± 0.04), and 4.7 cm (0.55 ± 0.03), as well as between 3.2 cm (0.45 ± 0.03) and 4.7 cm (all ps < 0.01). There was also a significant interaction between depth and stiffness, F(5.95, 65.41) = 2.29, p < 0.05, η² = 0.071: Both main effects enhanced each other. This means that a larger initial displacement increased the perceived softness more when the object's stiffness was low. This interaction might result from the underlying physical interaction: Stiff objects resist the initial displacement more strongly, they seem to deform less, and they return more quickly to their original shape; this means there is less time to observe the (already smaller) effect of the initial displacement. 

Figure 4

 

Results of the perceptual rating of the cube's softness in the motion cue condition. (A) Shows the mean ratings (averaged across participants) as a function of simulated stiffness of the cube and perturbation depth of the cylinder color-coded from hard (white) to soft (dark turquoise). (B) Shows the mean ratings as a function of stiffness split up for different depth levels. (C) Shows the mean ratings as a function of perturbation depth split up for different stiffness levels.

In sum, these results show that the simulated stiffness (in the range we chose for this experiment) does influence the perceived softness of the cube. The motion of the cubes seems to provide rich and diagnostic information about its softness. Additionally, the cubes appeared softer the larger the initial displacement, and even softer if both conditions (low stiffness and large initial displacement) were met. Thus, participants do not fully discount the magnitude of perturbation when judging stiffness, as would be required to judge the intrinsic properties of the material. As in the shape cue scene, the size of shape deformation contributed to the softness ratings. 

We can summarize the relative influence of the two physical parameters that we altered (intrinsic stiffness vs. perturbation depth) by fitting linear models to the data shown in Figure 3B and C and Figure 4B and C, and taking the slope as a measure of each parameter's influence on the participants' softness ratings. Figure 5A plots the slopes obtained for each participant's mean ratings per stiffness level (averaged across depth levels) on the x-axis against the slope obtained for the mean per depth level (averaged across stiffness values). Each ring represents a different participant. The slopes tend to cluster around different points for the two different conditions; in the shape cue scene only the perturbation depth shows a slope that is different from zero (note that one participant seems to have inverted the response scale), whereas in the motion cue scene both factors influence the perceived softness. Figure 5B shows the consistency within and between observers in the shape (bottom left) and motion (top right) cue condition. Overall, the consistency was very high. In the motion cue condition, the consistency between observers, defined as the correlation between their mean ratings in each condition, was on average r = 0.81 ± 0.10 (M ± 1 SD). Consistency within participants was defined as the average correlation between different runs of the experiment by the same participant. In the motion cue condition this was also high: The average correlation within participants was r = 0.79 ± 0.14. It was similarly high in the shape cue condition, on average r = 0.74 ± 0.16. The consistency between participants was lower in the shape cue condition (r = 0.59 ± 0.66). This was, however, mainly due to one participant who seemed to have used an inverted response scale (i.e., that person shows a high but negative correlation to all other participants—showing up in purple in the Figure). The correlation between all other observers (excluding Participant 8) was much higher, on average r = 0.88 ± 0.06. 

Figure 5

 

(A) Influence of stiffness and perturbation depth on softness perception. Open circles show data of individual participants in the shape (blue) and motion (turquoise) cue condition. For each participant we fitted a linear function to the mean ratings per stiffness level (averaged across depth levels) and per depth level (averaged across stiffness levels). Shown here are the slopes obtained from these fits plotted against each other. The two filled circles show the predictions for the slopes based on the mean deformation of the cubes (see Mesh analysis). (B) Consistency between (open cells) and within (bordered cells) participants for the shape cue condition (bottom left, purple-to-blue scale) and the motion cue scene (top right, purple-to-turquoise scale). Graphs color-code the correlation between average data of individual participants (interindividual consistency) and the average correlation between different runs of each individual (intra-individual consistency; cells with black borders).

Mesh analysis

Our results suggest that visual judgments of softness are strongly influenced by how much a shape deforms and how this deformation evolves over time. To further investigate the role of shape deformation in softness perception, we quantified the cubes' deformation in our different conditions. Since all our stimuli were computer generated, we have access to the mesh specifying the cube's shape in every frame (i.e., to the ground truth of the cube and its deformation). Each cube consisted of 1,352 vertices and 2,700 triangular faces. Only three sides of the cube were visible in our two scenes and we only considered the corresponding 721 vertices for our analysis. Note that the fact that a vertex lies on a visible side of the cube does not mean that this vertex was actually visible to the observer at all time points; it might have been occluded by the cylinder. However, assuming the operation of amodal completion mechanisms, we reasoned that the resulting shape would approximate the perceptual representation of the shape. Figure 6 shows the mesh of the undistorted cube as well as an example of a deformed cube. For each of the 721 vertices in the original cube, there is a corresponding vertex in the deformed cube. The length of the vector connecting two corresponding points in the original and the deformed cube is our unsigned measure of deformation. We calculated the deformation of each vertex in every frame compared to its corresponding vertex in the original undistorted cube. This tells us for each point in time how large the deformation was at each point in the cube. The mean deformation of a cube over time (i.e., averaged across vertices and frames) provides us with a simple shape-based prediction of the softness rating. A correlation of data and prediction was used to evaluate how well we could predict the softness perception with our simple model of deformation. 

Figure 6

 

Mesh analysis. The left image shows the three sides of the mesh underlying the original undistorted cube; the cube in the middle is an example of a deformed cube. The square on the right side shows enlarged the framed part of the mesh of the original and deformed cube superimposed together with the difference vectors between their vertices. The length of these vectors was used as the measure of deformation.

Shape-dominant condition

Figure 7A shows the deformation of each cube at each vertex point (and interpolated between them) averaged across time frames. In all conditions, the deformation was largest at points of the surface that had direct contact with the cylinder; the deformation spread from these points across the cube and was lowest close to the bottom. How each mesh deformed was mainly determined by the perturbation depth, not the simulated stiffness. This becomes even more obvious when looking at the mean deformation averaged across time and space, which is shown in Figure 7B. If observers based their judgments solely on the average deformation of the cube, this is how the pattern of ratings should look like. A comparison with Figure 4A shows that the deformation-based prediction closely resembles the pattern of perceptual ratings. Furthermore, the prediction of the slopes shown in Figure 5 is also very close to the true slopes of the participants. This is confirmed by the high correlation between the prediction and the average perceptual rating, r(23) = 0.95, p < 0.001 (see Figure 7C). Data of each participant was first normalized to a range between 0 and 1 (small dots in the figure) and then averaged (large dots in the Figure) to calculate this correlation coefficient. It is remarkable how well this simple model of deformation can predict the perceptual results. This confirms our earlier observation that the apparent softness of an object was determined by how much it deformed (rather than some accurate estimate of the intrinsic physical parameters), at least in our scene. 

Figure 7

 

Mesh deformation in the shape cue condition. (A) Shows the deformation of the cube color-coded at each vertex point (and interpolated between them) averaged across time as a function of simulated stiffness of the cube (x-axis) and perturbation depth of the cylinder (y-axis). For all cubes, the deformation was largest where the cylinder hit the surface and smallest close to the bottom where the cube was attached to the ground plane. Differences between the cubes appear larger between different depth levels than between different levels of stiffness. (B) Shows the mean deformation (averaged across all vertices in a cube and across time frames) for each condition. The horizontally striped pattern resembles the pattern of the rating data. (C) Shows the rating data plotted against the average mesh deformation. The 25 large dots show the mean rating for each condition (normalized for each participant and then averaged) and the mean mesh deformation (the values depicted in [B]). The small dots show the correlation between the prediction and the normalized data of each participant (i.e., 25 dots per participant).

Motion-dominant condition

Figure 8 shows the mean deformation of the cubes at each vertex point (and interpolated between them) averaged across time frames for this scene. The deformation was always lowest close to the bottom where the cube was attached to the ground plane and larger towards the top. How much each cube deformed on average was influenced by the simulated stiffness as well as the perturbation depth: The lower the stiffness and the larger the initial offset, the larger the deformation of the cube. The fact that some cubes appear completely blue in the Figure is due to the averaging over time; it does not mean these cubes did not deform at all, but that the stiffer cubes took on their original shape after few frames, thus influencing the average only little. As in the shape cue scene, the mean deformation averaged across time and space (see Figure 8B) resembles the pattern of the perceptual rating data very closely. There was indeed a high correlation between the deformation based model and the rating data, r(23) = 0.94, p < 0.001, although it appears that the model slightly underestimates the perceived softness. Nevertheless, it is notable how well the simple mean deformation predicts our data, given the fact that it is a simple average and ignores the specific motion patterns of the cubes. 

Figure 8

 

Mesh deformation in the motion cue condition. (A) Shows the deformation of the cube color-coded at each vertex point (and interpolated between them) averaged across time as a function of simulated stiffness of the cube (x-axis) and perturbation depth of the cylinder (y-axis). The cubes deformed more strongly in the upper half, because they were fixed to the ground plane. How much they deformed was influenced by the simulated stiffness as well as the perturbation depth (i.e., the initial offset). (B) Shows the mean deformation (averaged across all vertices in a cube and across time frames) for each condition. The pattern of the deformation-based prediction resembles the pattern of the rating data. (C) Shows the rating data plotted against the average mesh deformation. The 25 large dots show the mean rating for each condition (normalized for each participant and then averaged) and the mean mesh deformation (the values depicted in [B]). The small dots show the correlation between the prediction and the normalized data of each participant (i.e., 25 dots per participant).

Experiment 2: Optical cues

Experiment 1 measured how the pattern of deformation influences the perceived softness of an object, but it is also interesting to ask the extent to which the judgments are influenced by prior expectations based on the optical appearance of the surface. In Experiment 2 we took a subset of the stimuli from the previous experiment but changed their optical properties to resemble a range of different familiar materials (e.g., metal, wood, wax, or jelly), allowing us to test how much these contribute to the perception of stiffness of elastic objects. Different optical properties could provide different low-level cues or elicit different prior expectations that one might have about certain materials and material classes. 

Materials and methods

Stimuli

Figure 9 shows all different optical properties used in Experiment 2 rendered on an undistorted cube. This static image of an undistorted cube was used as a baseline condition to measure how soft the cubes appear based on optical information alone (i.e., to obtain the participants' priors). We chose 11 different optical appearances: the blue silicone-like material from Experiment 1 (silicone), a pinkish more translucent latex-like material, a red velvet- or suede-like material, reddish translucent gelatin-like material, red candle wax, light blue denim, matte green plastic-like material, wood, nickel, copper, and steel. To render the undistorted cube we took the first frame of the shape cue condition, but did not render the cylinder. The only other parameter we changed in comparison to the first experiment was the lighting in the scene; this was again a studio scene, but in was dimmer and more diffuse to enable more accurate rendition of a wider range of optical materials. The 10 new optical appearances were rendered onto the cube using the Maxwell (V.3.2.1.3; Next Limit Technologies, Madrid, Spain) render engine. 

Figure 9

 

Stimuli used in Experiment 2. (A) The images presented here show the static undistorted cubes with 11 different optical appearances that were used in the first part of the experiment. Part of the background was cropped for this illustration so that the cubes could be depicted larger. (B) Shows a sketch of the 25 versions of both scenes that were used in Experiment 1; highlighted are the two versions that were used in Experiment 2. These two combinations of stiffness and perturbation depth produce the largest differences in terms of the deformation of the cube. (C) shows some examples of cube in the shape condition. The images here show the last frame of the condition with the maximal deformation. With this deformation all four examples were rated similar in softness, although without any deformation steel was rated as appearing the hardest, velvet the softest and gelatine and nickel in between. Movies of the stimuli are available in the supplementary information.

In addition to the undistorted cube, we rendered two versions of the shape cue and the motion cue scene. In both scenes we rendered the cube with the maximum and minimum stiffness level and combined these with the minimum and maximum perturbation depth—that is, Stiffness Level 1 was combined with the maximum depth (3.0 and 4.7 cm, respectively) and Stiffness Level 5 was combined with the minimum depth (1.0 and 2.7 cm, respectively). These two versions were chosen because they represent the maximum and minimum in terms of mean deformation and they were rated differently in both the motion and the shape cue scene of Experiment 1. For the motion cue condition we were only able to render six of the 11 optical materials, due to texture-mapping related artefacts that occurred for all textured materials—that is, we rendered latex, wax, gelatin, nickel, silicone, and plastic. This was sufficient for our study, since we were not interested in measuring responses to specific optical materials, but we rather wanted to cover a range of different apparent stiffnesses. All stimuli are available for download at https://doi.org/10.5281/zenodo.155310. 

Participants

Twelve students from the University of Gießen participated in Experiment 2, six males and six females. Observers were on average 25 ± 4.6 years old (M ± SD). They were naive regarding the specific aims of the study. Written informed consent was obtained from every participant before the experiment. The procedure was in accordance with the Declaration of Helsinki and approved by the local ethics committee LEK FB06 at Gießen University. 

Procedure

Before the start of the experiment, the participants were instructed that they would see images and animations of different cubes and would have to rate the softness of these cubes. Softness was explained as how easy it would be to squeeze or push in the cube (using the German word eindrücken). Unlike in Experiment 1, no real physical objects were used in the instruction phase to avoid any influence on the stiffness ratings of the similar-looking blue stimulus in the main experiment. The experiment consisted of three blocks. In the first block, participants were presented with images of the static undistorted cubes. On each trial subjects provided a rating as in Experiment 1, by continuously adjusting the position of a dot on a rating scale. This condition was always completed first, because we were interested in the appearance of the cubes with respect to the optical properties irrespective of shape or motion cues (i.e., in the observers' priors). Before the rating, all cubes were presented once to the observers in random order so that they got an impression of the total range of materials we covered (no response was required during the prepresentation phase). 

The following two blocks were ratings of the shape cue and the motion cue condition (order counterbalanced across participants). Participants were first presented with four example stimuli of different materials (randomly chosen) and both deformation levels (randomly assigned). During the prepresentation phase, no responses were required. During the main experiment, on each trial, participants were presented with the animations playing in a loop until a response was given. For the shape cue condition, the frames were looped forwards and backwards as if the cylinder intruded then retreated from the cube, to yield a more natural cycle without abrupt transitions. As in Experiment 1, the response was given by adjusting a slider on the scale between soft and hard with the mouse, and confirming the judgment with a mouse click. Three repetitions were obtained for each combination of scene and material in the second two blocks. The experiment took approximately 15 min. Raw data from all experiments can be downloaded here: https://doi.org/10.5281/zenodo.155310. 

Results and discussion

The results of Experiment 2 are shown in Figure 10. Figure 10A shows the average softness ratings for the static cubes rendered with different optical properties and ordered accordingly. A couple of observations can be made here: First, the different surfaces evoked substantially different expectations about the softness of the cube. This was confirmed by a significant main effect of the surface in a one-way repeated-measures ANOVA, F(4.99, 54.85) = 18.36, p < 0.001, η² = 0.577). Second, our aim was to choose optical properties that cover a broad range of expectations from hard to soft with many steps in between. This was indeed what we found. It should also be noted that although we label the different optical appearances to make it easier to describe the stimuli and results, these labels do not necessarily correspond to how the participants would label the materials. However, for the purpose of this study, it is not relevant whether they recognized, for example, what we labeled copper as being copper; it only matters that some optical materials were perceived as hard and others as soft, which was the case. 

Figure 10

 

Results of Experiment 2. (A) Ratings of the undistorted cubes with different optical appearances. Bars represent the mean ratings (averaged across participants) together with the standard error of the mean between participants. Data is ordered from hard to soft according to the mean ratings. Dark gray bars show materials that were subsequently used in the motion and shape cue condition; light gray bars show materials that were only used in the shape cue condition. (B) Ratings of the distorted cubes in the second version of the motion and shape cue condition averaged across all optical appearances (M ±1 SEM between participants). (C) Mean ratings of the deformed cubes of varying optical appearances as a function of the ratings of the same but undistorted cubes. Cue and deformation are color-coded as in (B). Each dot shows one optical appearance ordered according to the rating of the undistorted cubes as in (A).

In sum, depending on their different optical appearances the static cubes were perceived as being more or less stiff. In a second step we wanted to test whether their surface properties also influence softness perception in the presence of deformation cues. We used three different analyses to answer this question: two types of ANOVA and a regression analysis. First, we calculated a 2 (cues: motion vs. shape) × 2 (deformation: maximal vs. minimal) × 6 (optical appearance) repeated-measures ANOVA in which we included only those six optical appearances that were used in the motion as well as shape cue condition (i.e., all those without texture). This ANOVA revealed three significant effects: Unsurprisingly, there was a significant and large effect of the deformation on the softness rating, F(1, 11) = 218.79, p < 0.001, η² = 0.893; the cube with the maximal deformation was always perceived to be softer than the other one (see Figure 10B). This is basically a replication of the results from Experiment 1. We also found a small but significant interaction between the deformation and the available cue, F(1, 11) = 7.10, p = 0.022, η² = 0.009; the difference between the ratings for the maximal and minimal deformation was larger for the motion cue condition than for the shape cue condition (see Figure 10B). Furthermore, and more central to our research question, we also found a significant interaction between deformation and optical appearance, F(2.27, 25.01) = 18.02, p < 0.001, η² = 0.280. However, this interaction seemed to be unsystematic. No other main or interaction effect was significant—that is, there was no main effect of the optical appearance, F(1.62, 17.78) = 0.41, p = 0.842, η² = 0.014; there was no general difference in the ratings for the two cues, F(1, 11) = 0.01, p = 0.915, η² = 0.000; no interaction between the two, F(2.15, 23.69) = 1.29, p = 0.283, η² = 0.021; and no three-way interaction, F(3.15, 34.60) = 2.05, p = 0.122, η² = 0.022. 

Second, we ran a 2 (deformation) × 11 (optical appearance) repeated-measures ANOVA including all 11 optical appearances on the data from the shape cue condition. We again found a large significant main effect of deformation, F(1, 11) = 108.44, p < 0.001, η² = 0.816, and no main effect of the optical appearance, F(4.80, 52.79) = 2.31, p = 0.059, η² = 0.062. In contradiction to the previous results there was no interaction between optical appearance and deformation, F(4.49, 49.42) = 1.02, p = 0.430, η² = 0.020. This is surprising given the fact that the more extreme appearances were included in this but left out in the previous ANOVA (i.e., the materials that were rated as softest/hardest regarding the static cube). 

In sum, these first two analyses show that the perceived softness depends on the deformation of the cube; that the cube with the smaller deformation is perceived harder than the other one; and that the difference between the two seems to be larger in the motion cue scene. There was never a main effect of the optical appearance on softness perception and only once an unsystematic interaction with the deformation. This suggests that when deformation cues are available the optical appearance of the cube made only a weak and erratic contribution to perceived stiffness. The nonsystematic nature of the interaction suggests that the optical properties modulate the visibility of the motion and shape cues that provide the primary sensory data for the perception of stiffness, rather than directly indicating a specific stiffness value themselves. In other words, by varying the materials we not only varied the high-level expectations associated with them (e.g., latex is soft vs. steel is hard vs. wax is intermediate) but also varied the low-level properties of the materials (e.g., local contrast, texture, translucency, etc.). Because the variations in low-level properties were unsystematic, we did not expect any effect of the amount of texture, translucency, or contrast on softness perception. However, these low-level features might have influenced the perceived deformation of the shapes. For example, where the material yields lower local contrast, the optic flow information will be sparser, leading to more noisy estimates of the motion cues, and a correspondingly weaker estimate of the softness. Similarly, translucency or texture may facilitate the detection of deformation within the object and along its contours, as suggested by previous studies (Han & Keyser, 2016; Kawabe & Nishida, 2016). Due to our experimental design, such low-level influence would lead to an unsystematic interaction between optical properties and deformation as we found in our data. 

Figure 10C shows average ratings for the deformed cubes in the motion cue and shape cue conditions as a function of their ratings when not deformed. Different colors show different cue/deformation conditions. Each point shows the average for one optical appearance; the materials are ordered according to their rating of the nondeformed cube (i.e., as in Figure 10A). If there was a systematic influence of the optical appearance on the perceived softness, this relationship should (in the simplest case) show up as a slope in this plot. This is, however, not what can be observed. Statistically this was confirmed by using the rating of the nondeformed cubes as a predictor of the rating of the deformed cubes in all four conditions and for each participant individually (i.e., four regressions per participant). The steeper the resulting slopes, the larger the influence of the optical appearance on the perceived softness. However, the linear fits had only a very small slope, 0.03 ± 0.12 (M ± SD) and were on average not different from zero in any of the four conditions (all ps > 0.05). Also the variance explained by the rating of the nondeformed cube was low, on average 21.83% ± 24.19% (M ± SD). We therefore concluded that if no other cues are present, the optical appearance of an object influences whether the object is perceived as rather soft or hard. However, this influence essentially vanishes as soon as deformation cues are available. The larger this deformation is, the softer the object is perceived. This suggests that the optical appearance acts like a low-reliability sensory signal, or as a prior, whose influence is swamped once more diagnostic sensory data (shape deformations) are present. 

General discussion

In this study we investigated the role of three possible cues the human visual system could use to infer the stiffness of elastic objects: shape, motion, and optical cues. Physically there is no link between the stiffness of an object and its optical appearance. However, through a lifetime of previous experience with different materials we might have learned associations between certain optical properties and internal features of objects such as its stiffness. In Experiment 2 we saw that when the objects are static and undistorted, observers do have expectations about their softness based on their optical properties. In principle such prior expectations could also influence the perception of deforming cubes—for example, in terms of an integration with the deformation cues, similar to classic cue combination effects (e.g., Ernst & Banks, 2002; Hillis, Ernst, Banks, & Landy, 2002; Landy, Maloney, Johnston, & Young, 1995; Weiss, Simoncelli, & Adelson, 2002). Alternatively, there could be contrast-like interaction between material priors and sensory input as in the material-weight illusion (Buckingham, Cant, & Goodale, 2009; Charpentier, 1891; Ellis & Lederman, 1999; Paulun, Buckingham, Gegenfurtner, Fleming, & Goodale, 2015). However, we seem to find neither of these effects; instead, optical cues seem to be essentially ignored in the presence of deformation cues. In the framework of Bayesian cue combination, this would be consistent with the idea that the optical cues are simply very unreliable compared to the deformation cues, and therefore have a negligible influence on the resulting percept. 

It is also interesting to note that in some cases the optical appearances interacted with the cubic shape in complex ways, yielding nonhomogeneous appearances. For example, the denim stimulus looks like a hard block that has been very tightly coated with denim, or has a “denim pattern” veneer coating. Similarly the metal block could be perceived to be hollow, potentially altering its expected mass, flexibility, and other attributes. Such nonhomogeneous impressions should be the topic of future studies. Little is known about how texture, optics, and shape properties are integrated to determine high-level physical properties of objects—these results hint at some potentially interesting nonlinear interactions (i.e., whether a given optical appearance indicates a soft or hard material may change completely depending on the shape to which it is applied). 

In principle, there could also be a low-level influence of the optical appearance on softness perception, since certain surface properties like texture or translucency could facilitate the perception of deformation—as has been suggested by previous research (Han & Keyser, 2015, 2016; Kawabe & Nishida, 2016). Such effects might be responsible for the small and unsystematic interactions between optical appearance and cue type that we observed. However, our main finding is that when deformation cues are present, optical appearance has a negligible influence on softness perception. This is in line with recent findings that shape and motion cues are used to infer fluid viscosity whereas the influence of optical cues is negligible (van Assen & Fleming, 2016). Different results were found in the domain of cloth perception: Aliaga, O'Sullivan, Gutierrez, and Tamstorf (2015) showed that in most cases the perception of cloth categories is dominated by the optical appearance of the cloth rather than its dynamics. However, their task was quite different from ours. In their study, observers were presented with combinations of optical appearance and dynamics from different classes of cloth and were asked to match a given material—here cloth categories like cotton or silk. In our task, by contrast, observers were not asked to classify materials, but to estimate a certain material property, specifically stiffness. The dynamics in our experiments were not matched to specific materials, only the optical properties were. If we had asked our participants to identify classes of materials such as wood, wax, or metal (instead of estimating stiffness), they presumably would have relied more on optical properties, similar to the results of Aliaga and colleagues. In line with our results, observers make use of dynamic information when estimating specific mechanical properties of cloth, such as mass and bending stiffness (Bi & Xiao, 2016; Bouman, Xiao, Battaglia, & Freeman, 2013). When making these estimates, observers show a fair amount of constancy across different variations of an external force, which is in congruence with the optic flow patterns of the cloth (Bi & Xiao, 2016). 

The other experiment presented here focused on two scenes in which either the shape or the motion cues were most dominant (Experiment 1). What both scenes have in common is the deformation of the cube over time. However, it is important to note that in the shape-dominant condition, there is little reason to think that motion per se contributed to the perception of stiffness, because each frame was essentially a steady-state response to the instantaneous deformation of the cube. This suggests that playing the movie at different speeds, for example, would have little effects on apparent properties of the cube. In contrast, for the motion-dominant condition, the motion resulted from the reverberation of the object, whose precise form and temporal envelope was determined by the material's intrinsic properties. This might explain why stiffness ratings were so much more closely related to the physical stiffness for the motion cue than for the shape cue scene. 

For both scenes, the magnitude of deformation turned out to be a very good predictor of the reported softness of the cubes. In the shape cue scene, the mesh deformation was almost exclusively determined by the perturbation depth of the cylinder—rather than by the cube's stiffness, and so were the ratings. That suggests that the deformation cue dominated judgments even when it was not veridical: The more the object deformed the softer it appeared, irrespective of whether this deformation was due to the object's stiffness or some other factor. Another way of putting this is that when asked to judge stiffness in these scenes, participants resorted to a simple heuristic based on the extent to which the object tended to change shape in response to external perturbations. For many common scenes, this is not an unreasonable approach to judging stiffness, as it correlates with the physical property of interest in many cases. Moreover, it is open to debate whether the aim of the visual system is even to estimate intrinsic physical parameters of materials per se: From a functional point of view it might be more useful to operationalize stiffness as “the extent to which an object deforms in response to perturbation”—in the same way as glossiness, rather than being an estimate of the specular reflectance of surfaces may instead be “the extent to which a material exhibits highlights” (Fleming, 2012, 2014). 

If we do treat stiffness perception as a process of estimating physical parameters of materials, then the deviation from veridicality is an important observation since it also shows that observers did not show stiffness constancy in this specific scene. Everyday experience tells us that, in general, humans likely exhibit some degree of stiffness constancy—that is, the ability to estimate stiffness across differences in shape, size, optical appearance, and external forces. However, so far it is not clear to which extent we are stiffness constant or which cues are diagnostic for stiffness and invariant across a wide variety of scenes and objects. For example, it is an open question whether stiffness impressions derived from a bouncing object (where mesh deformations are small compared to the overall trajectory of the object) can be successfully matched to objects like ours that are deformed without changing position. Here, we show that deformation is highly diagnostic for stiffness, but not invariant across different scenes. In the shape cue scene this leads to a failure of stiffness constancy. However, the external force in the animation used here is highly ambiguous; the cylinder floats through the air, driven by some unknown force. In real life, conditions might often be less ambiguous because the force is applied by the observers themselves, or can be approximately estimated from other cues. In these cases the deformation might actually be more informative and stiffness constancy higher. It is also worth noting that the failure of stiffness constancy in our scene is also somewhat unsurprising in the sense that there was very little image data to distinguish between the different materials: The change in shape was almost identical irrespective of the stiffness. Observers showed much more stiffness constancy within the motion cue scene, but because of the experimental design, we cannot compare constancy across scenes. In Experiment 2 observers showed a larger amount of stiffness constancy over a range of different optical appearances as well as two different scenes, and in both cases they must have relied on deformation information to accomplish this. Future research should investigate more systematically how constant observers are with regard to stiffness and how exactly this constancy is achieved. Scenes (as the shape cue scene) in which the perception is erroneous and constancy low, might actually be helpful in understanding the underlying mechanisms—a good model should be able to account for both the successes and failures of stiffness perception. 

It should also be noted that participants were only asked to rate stiffness relative to the specific range of stimuli that they observed; their judgments were not estimates of absolute stiffness on some fixed, universal scale. This is important because in order to estimate absolute stiffness our observers would have had needed additional information about the underlying geometry and physics of the scene (most notably, size, mass, and forces). While we cannot make direct inferences about how such quantities are perceived in our experiments, there are still some relevant observations. There are several potential possible cues that may have influenced observers' impressions of size: (a) the viewing angle, (b) the translucent appearance of the object (Experiment 1), and (c) the retinal motion speeds of the object (motion cue scene). In our animations, the observer views the object diagonally from above as would be typical if viewing a small object on a table. Thus, the viewing angle may have contributed to the perception of a small, rather than large, object. Moreover, for translucent objects, the amount of light leaving the object after subsurface scattering depends not only on the internal properties of the material but also on the size of the object, so subsurface light scattering can be informative about the size of objects (Fleming & Bülthoff, 2005; Jensen & Buhler, 2002). Large objects have to be extremely translucent to appear so. Thus, the translucent look of our objects may have contributed to the impression of a small size. Furthermore, in the motion cue scene, the size and mass of the object would influence the observed retinal motion patterns. Movement in the presence of gravity varies as a function of scale. For a given angle of view, large objects tend to be associated with slower retinal motion speeds (consider a falling tree vs. a matchstick). Similarly, very large elastic forces would have to be involved to evoke the rapid motions observed in the animations if the object was large. How motion cues contribute to perceived scale and distance is a fascinating topic for further research. 

Another implicit inference that needs to be made when judging stiffness is of course the contribution of the external force. This becomes especially apparent in the shape cue condition. Our scene is very ambiguous (probably more than most real-world scenes, in which additional cues may be available) because the cylinder is floating in the air, driven by an invisible force. The judgments of the participants in our study are consistent with the assumption of a constant force in all animations. This is in line with the bias that was found in a study by Fakhourny et al. (2015). They showed that observers are more accurate in judging the softness of objects when they were indented with the same amount of force rather than to the same depth. Both findings imply that in an ambiguous scene, observers are more likely to assume that a constant force is used across different versions of that scene. This is an interesting observation especially because humans themselves do not seem to use a constant force when exploring the softness of different objects haptically, but appear to adjust the force depending on the stiffness of the target object (Kaim & Drewing, 2009). In haptic perception it has also been shown that compliance estimates are more accurate for deformable objects (e.g., rubber), than compliant objects with a rigid surface (e.g., a spring cell), because for any given force, the pattern of deformation across the skin provides additional information about compliance (Srinivasan & LaMotte, 1995). It would be interesting to speculate if such differences also appear in the visual domain (through different mechanisms, of course). It may be that deformable objects provide more visual cues through how they bend and curve. On the other hand, the external force may be represented more accurately in the direct mapping between force and height of the compressed object. 

Although we find that deformation is a good predictor of visual stiffness ratings in our experiments, there are some important caveats. First, it should be noted that we do not suggest that the mesh deformation quantity that we used is directly computed by the human visual system. It seems quite unlikely that the visual system measures the difference in shape between the deformed and nondeformed versions of the object: Indeed, we can derive a vivid impression of a deformation even when the original, nondeformed object has never been seen (e.g., the final frames of the shape cue conditions, as shown in Figure 2A). Presumably the visual system has some way to estimate deformation that does not require computing deviation from a known reference object. Thus, the deformation measure should be viewed as a way of characterizing the information available, rather than a process model of stiffness perception. 

Second, it should be noted that although deformation correlated very well with ratings, in both experiments there was a clear nonlinear relationship between deformation magnitude and responses. Importantly, however, the curves bow in opposite directions, meaning that there is no single fixed nonlinear mapping from deformation magnitude to perceived stiffness: Clearly something more complex is taking place. 

Third, in this study, we focused on the absolute magnitude of the deformation averaged spatially as well as temporally. The deformation magnitude is appropriate here, because of the rather simple way the scenes are set up. However, for a broader range of scenes and situations this measure would clearly be insufficient, as it would even count rigid transformations (translation, rotation) as deformations. This could be overcome by first subtracting any rigid component of the motion and only measuring the remaining nonrigid deformation. However, it is also clear that the relevant deformation happens within the object and a more sophisticated measure should take this spatial requirement into account. 

Fourth, although our data suggests that the magnitude of deformation is the dominant cue in softness perception, as just indicated, other properties of the deformation are presumably also important. Not all nonrigid deformations are interpreted as being due to the material's stiffness. For example, adding random noise to an object's relief, or morphing between a bunny and an elephant are both nonrigid shape deformations, but they most likely do not elicit a compelling impression of elastic stiffness. Indeed, probably only a very restricted and specific subset of nonrigid deformations is interpreted as elasticity. Further work is required to identify the specific cues that are required. With regard to the shape, this might be, for example, the curvatures and how the elastic object bends around another one (like the cube around the bottom half of the cylinder). With regard to the motion, the cues might include specific motion characteristics (e.g., frequency or damping of periodic motions). 

Furthermore, it is also clear that in many cases it is not even necessary to observe the process of deformation to judge an objects' stiffness. Figure 11 shows an object that has settled under gravity into a specific shape. Without observation of how the object got there and without previous experience with that specific object or its material, it is possible to infer that it has deformed and even to derive a fairly clear impression of its stiffness. It is another interesting topic for future research to investigate how the visual system can infer material properties such as stiffness, as well as the original undistorted shape, and potentially also the process that formed the new shape, from just a single snapshot. 

Figure 11

 

A soft object has settled under its own weight. Without observation of the deformation process, the object, or its material it is easy to make some inferences about its material properties.

In sum, elastic objects deform in distinctive ways in response to an external force. How much they deform depends on their stiffness as well as the amount of force they are responding to. In order to visually judge this internal property (stiffness) the human visual system seems to make use of the magnitude of deformations. The more an object deforms, the softer it appears, even if this percept is not veridical. This is in line with previous research on the visual perception of fluid viscosity (Paulun et al., 2015). The visual system may not accurately simulate the physics behind the deformation (i.e., the internal properties of the object and its response to the external forces) in order to interpret the observed shape. Instead, it might use a statistical appearance model (Fleming, 2014)—namely a low-parameter internal model of the primary degrees of freedom across observations of deformed objects. The magnitude of deformation could be an important characteristic in the feature space of such an internal model of nonrigid objects. Future research should investigate more of the features that the visual system uses when judging deformable materials and whether these features are the same as those used to make predictions about future states (e.g., to guide actions). 

Acknowledgments

We would like to thank Eugen Prokott for assistance in designing the materials and Maarten Wijntjes for helpful comments on a previous version of this manuscript. This work was supported by the German Research Foundation (DFG) International Research Training Group (IRTG) 1901 “The Brain in Action—BrainAct,” the DFG Collaborative Research Centre SFB-TRR 135 “Cardinal Mechanisms of Perception,” the European Union (EU) FP7 Marie Curie Initial Training Networks (ITN) Project PRISM (Perceptual Representation of Illumination, Shape and Material; PITN-GA-2012-316746) and the European Research Council (ERC) Consolidator award “SHAPE” (ERC-CoG-2015-682859). This publication represents a component of the doctoral (Dr. rer. nat.) thesis of V. C. Paulun in the Faculty of Psychology and Sport Science at the Justus-Liebig-University Gießen, Germany. 

Commercial relationships: none. 

Corresponding author: Vivian C. Paulun.