What change detection tells us about the visual representation of shape

Abstract

Many recent findings suggest that human observers are surprisingly “blind” to changes in visual displays, failing to notice when substantial scene elements are added, subtracted, or altered in successive presentations of the scene. But observers are far more sensitive to certain visual changes than others, and we suggest that which types of changes enjoy differential sensitivity can reveal a great deal about the underlying visual representations. In this study, we investigate how the human visual system represents the shape of objects by demonstrating a previously unknown influence on detection of changes in shape: the sign of contour curvature. We show that subjects are substantially more sensitive to changes in concave regions of a shape’s contour than to changes in convex regions, even when these changes do not alter the number or location of parts. Further, we show that this effect is modulated by figure-ground assignment, so that changes to the same physical contour are more or less detectable, depending on the contour’s perceived figural status, which determines whether the change falls in a concave or convex region. The results demonstrate a heightened sensitivity for changes at concavities that is not reducible to a sensitivity to changes in gross part structure.

Introduction

The representation of shape is one of the most fundamental problems in the study of the human visual system. Yet our understanding of how shape is mentally represented remains surprisingly incomplete. From a geometric point of view, shape involves many distinct but interrelated parameters, leading in principle to a large (perhaps infinite) number of potential representational schemes. But the human visual system elevates certain parameters above others, choosing to encode or emphasize these while suppressing or deemphasizing others—a choice that in turn determines how any given shape will be encoded and perceived. However, we lack a thorough understanding of what aspects of shape are particularly important, and thus how shape is actually represented by the visual system. This study seeks to shed light on this fundamental problem by applying an existing methodology in a novel way; the results help reveal which shape parameters play especially prominent roles in mental shape representation. 

Change blindness and differential sensitivity

Many recent studies have demonstrated that human observers are surprisingly insensitive to changes occurring in alternating visual displays—an effect sometimes referred to as change blindness (e.g., Simons & Levin, 1997; Rensink, O’Regan, & Clark, 1997). The surprise in these studies comes from the fact that observers regularly miss not only changes to small details of an image but also some that seem “large” and “meaningful” (e.g., the identity of a speaker or the presence of a building). Henderson and Hollingworth (2003) recently demonstrated a blindness to change even in cases where—by shifting multiple occluders in unison—the entire image was altered from one viewing to the next (i.e., every image pixel was changed from one view to the next). Yet some types of changes are presumably more detectable than others. We suggest that, as far as revealing the underlying form of visual representations is concerned, it is sometimes more useful to examine differential sensitivity to different kinds of changes applied to some well-specified class of stimuli. As emphasized by Marr (1982), the visual system cannot represent all of the information in the visual array equally. Rather, any representational scheme must make certain features more explicit at the cost of others. It is within this “representational language” (see Feldman & Richards, 1998) that differences in detection abilities can be expected to be manifest. Many traditional psychophysical methods involve measurement of sensitivity to stimulus differences along specific, targeted, dimensions (e.g., brightness and length). But in changedetection experiments, the change on each trial can come along any dimension; the observer generally has no fore-knowledge of which aspect of the stimulus array it will involve. Hence these tasks are uniquely suited to determining which—among the many dimensions that could be represented—actually are represented. In this way, change detection can be viewed as a means of surveying stimulus dimensions and estimating their relative explicitness and emphasis in the underlying visual representation. 

In this work, we apply this logic to the experimental study of visual shape representation. In particular, we measure differential sensitivity to changes involving shape properties, while controlling the magnitude of stimulus change itself. If one property is more central to shape representation, then we can expect that changes to that property will be detected more readily. Conversely, we can expect observers to be relatively insensitive to changes in properties that are less prominent in their shape representations. In the following experiments we examine the extent to which a specific shape property—the sign of contour curvature—leads to differential performance in a change detection task. 

Shape representation: Magnitude and sign of curvature

In attempting to explain how visual descriptions tend to minimize redundancies and maximize economy, Attneave (1954) observed that points of high curvature along contours appear to carry the greatest information. The importance of curvature in shape representation has since been corroborated empirically (Singh & Fulvio, 2005; De Winter & Wagemans, 2004; Norman, Phillips, & Ross, 2001; Wolfe, Yee, & Friedman-Hill, 1992), as well as mathematically in terms of a formal information measure (Feldman & Singh, 2005). In addition, the human visual system has been found to be extremely sensitive to the magnitude of curvature, even showing hyperacuity effects (Watt & Andrews, 1982; Wilson & Richards, 1989; Wilson, 1985). 

Attneave’s original observation concerned only the magnitude of curvature, but more recently much research has recognized the psychological importance of the sign of curvature as well. (Curvature is designated positive when the contour is turning toward the inside of the shape, i.e., in convex regions, and negative when turning away from it, i.e., in concave regions.) Hoffman and Richards (1984) and Koenderink and Van Doorn (1982) suggested that regions of positive and negative curvature tend to play different roles in shape representation. In particular, Hoffman and Richards (1984) proposed that negative minima of curvature—points with locally maximal magnitude of curvature that lie in concave regions—define boundaries between perceived parts (whereas otherwise similar points in convex regions do not have such status). 

The perceptual consequences of these asymmetric roles played by positive and negative curvature in part segmentation have been demonstrated in a number of contexts, including figure-ground perception (Baylis & Driver, 1995b; Driver & Baylis, 1996; Hoffman & Singh, 1997), amodal completion (Liu, Jacobs, & Basri, 1999), memory for shapes (Braunstein, Hoffman, & Saidpour, 1989), the perception of symmetry and repetition in visual patterns (Baylis & Driver, 1994, 1995a), the localization of vertex height (Bertamini, 2001), the perception of transparency (Singh & Hoffman, 1998), and visual search (Wolfe & Bennett, 1997; Hulleman, te Winkel, & Boselie, 2000; Xu & Singh, 2002). Moreover, a number of recent studies have demonstrated that selective attention can be allocated to individual parts (Barenholtz & Feldman, 2003; Vecera, Behrmann, & Filapek, 2001; Vecera, Behrmann, & McGoldrick, 2000; Watson & Kramer, 1999). In particular, visual comparisons are found to be systematically faster and more accurate when they involve features of a single part of an object, rather than features found on two distinct parts—even when the curvature profile of the intervening contour is carefully controlled (Barenholtz & Feldman, 2003). 

A plausible overall account of the above findings is that the visual system’s shape representation is built around something like a “part skeleton” (Blum, 1973; Kimia, Tannenbaum, & Zucker, 1995). Such a compact representation emphasizes qualitative structural properties, such as the number and location of parts, rather than fine metric details, with one consequence being that subjects are especially sensitive to shape changes that qualitatively alter this part organization (Keane, Hayward, & Burke, 2003). Indeed, using a change detection task, we recently found that subjects are especially sensitive to shape changes that involve the removal or introduction of a new concavity along the contour, which change the total number of parts or axial branches on the shape, compared to those that involve a new convexity, which do not (Barenholtz, Cohen, Feldman, & Singh, 2003). Changes to the number of concave vertices were detected with more than twice the accuracy (d′) than changes to convex vertices. These findings highlight the importance of part decomposition in shape perception, and the primacy of negatively curved contour regions, at least insofar as they contribute to determining qualitative organization into parts. 

Is the increased sensitivity to changes involving concavities solely due to representational prominence of gross part structure (and hence heightened sensitivity to changes that alter this part structure), or is it due to a fundamental representational asymmetry between concavities versus convexities themselves? In the experiments presented here, we test whether subjects will exhibit a heightened sensitivity to shape changes in concave regions that do not alter the number or location of parts. A concavity or convexity on randomly generated polygonal shapes was either enhanced or diminished by slightly moving a single vertex of the shape. By design, these shape changes were subtle. In particular, because the changes never resulted in the introduction of a new concavity or convexity, the number of parts and their gross spatial relations were not altered. Thus, any observed advantage for detecting changes to concavities cannot be attributed simply to a heightened sensitivity to the number and location of parts. If superior detection for concave changes is observed, it would suggest differential sensitivity to negatively curved regions. 

It should be noted that, although the shape changes used do not alter qualitative part structure, they of course affect more subtle “metric” properties of parts and their axes (e.g., the salience of part boundaries; Hoffman & Singh, 1997; Singh & Hoffman, 2001), the length of axial branches, and the thickness of the parts around an axial branch (Biederman, 1987). Indeed, every change to the geometry of contour necessarily induces some metric change to a part or an axis (see Figure 1). However, in our experiments such metric changes are the same in the convex and concave case, and moreover in Experiment 2 are carefully constructed to be completely local (i.e., only altering the contour in a small neighborhood of uniform sign of curvature). As our changes do not alter gross part structure, and change metric shape equivalently in concave and convex cases, a superior sensitivity to concave changes in the current experiments would demonstrate a fundamental role for concave contour segments in the representation of object shape. 

Figure 1

 

Any change to a contour necessarily alters some metric shape properties. But some changes (lower left) also alter the gross part structure (e.g., the topology of the shape skeleton), whereas a similar change at another location along the contour may not (lower right). Our experiments use changes of the latter kind to investigate representational asymmetries arising from the sign of curvature.

Experiment 1

Stimuli consisted of two brief successive presentations of a randomly generated “nonsense” polygonal shape, separated by a mask. In change trials (50%), a single vertex on the polygon was displaced between presentations. Displacement involved the repositioning of the vertex by moving it a set distance, either inward or outward, under constraints described below. Two experimental variables were manipulated: change type (concave vs. convex, referring to the contour polarity at the changed vertex) and change direction (enhancing or decreasing the “sharpness” of the vertex). The subject’s task was to indicate whether or not a change had occurred between the two presentations of the shape. 

Method

Observers

Thirteen Rutgers undergraduates served as naive observers for course credit. 

Stimuli

Stimuli were computer-generated filled polygonal shapes measuring between 2.4 and 4.8 deg of visual angle in both height and width. The shapes were generated in sets consisting of a base shape and four modified versions of that shape, to ensure that a change consistent with each experimental condition was possible for every shape presented. The base shape was generated by choosing between 9 and 12 points, each located at a random distance (between 1 and 2.5 deg of visual angle from the center of the screen) along successive radial axes (separated by 30° to 40° of polar angle) projecting from the center of the screen, and joined by straight line segments. The changed shapes were then created by displacing a single vertex, which was the fulcrum of either a convex or concave angle, depending on condition, under the following constraints. 

In creating the changed shape, our goal was to ensure that the changes did not qualitatively alter the part structure of the base shape. This is tricky because every change to the position of a vertex of a polygon necessarily alters not only the angle at that vertex but also the angles at the two adjacent vertices. If not constrained, a large enough change in the position of one vertex could, for example, transform a neighboring convexity into a concavity. Hence we needed to ensure that the perturbation of one vertex did not change the sign of curvature at either the vertex in question or either of its neighbors. We did this by first choosing a small displacement magnitude, selected at random from between 5 and 30 pixels (3.6–21.6 arcmin). The vertex in question was then translated through this distance in the normal direction (defined by the angle bisector between the two adjacent segments) inward or outward, depending on the condition. Then the three altered vertices were tested geometrically to ensure that none of their signs of curvature had changed. If one had, the shape was rejected, and a new random shape was generated and altered. This process was repeated until an acceptable base shape and its respective altered shapes were found. Each trial presented a unique polygonal shape. A new set of shapes was generated for each observer. Figure 2 illustrates typical shape changes. 

Figure 2

 

Types of shape changes in Experiment 1.

Design and procedure

On each trial, the observer was presented with the following sequence (see Figure 3): (1) a fixation cross presented for a variable duration between 300 and 700 ms; (2) the first shape stimulus for 250 ms; (3) a mask for 200 ms; (4) the second shape stimulus for 250 ms; and (5) the mask until response. Two independent variables were manipulated: change type (concave/convex), and change direction (enhance/diminish). Thus, in the change trials, four types of changes were possible (see Figure 2). In the no-change trials, the same shape (either the base shape or a changed version) was simply presented twice. 

Figure 3

 

Trial sequence for Experiment 1.

The observer’s task was to indicate whether or not a change had occurred between the two presentations of the shape. They responded using the keypad, with feedback provided by a beep on incorrect responses. 

Each observer ran eight experimental blocks for a total of 768 trials. Each block contained 96 trials (48 change and 48 no-change). This number allowed for the crossing of the two experimental variables (2 × 2) in change trials, as well as balancing for the number of concavities and number of sides in the base shape (in both change and no-change trials) within each block. 

Results

Proportion correct was much higher for concave changes (mean = 63.89%, SE = 3.13%) than convex changes (mean = 45.78%, SE = 2.94%). Data were converted to d′ for analysis of variance (concave mean = .92, SE = .10; convex mean = .42, SE = .05; see Figure 4). The overall difference between concave and convex change was highly significant, F(1,12) = 33.94, p < .0001. Enhancing changes were marginally more detectable overall than diminishing changes, F(1,12) = 4.575, p = .054. 

Figure 4

 

Results of Experiment 1. Error bars indicate standard error.

Discussion

The results of Experiment 1 provide strong support for the importance of sign of curvature in shape representation. Observers were far more sensitive to shape changes affecting concavities than to corresponding changes affecting convexities. This differential sensitivity was observed despite the fact that all changes preserved the sign of curvature throughout the shape—and therefore preserved the shape’s qualitative part structure. Thus, the heightened sensitivity at concavities cannot be attributed simply to a sensitivity to changes to overall part organization. It is not necessary for a part to appear or disappear, or any other similarly qualitative change (such as those in Barenholtz et al., 2003), for the change to be especially detectable. 

Experiment 2

In Experiment 1, the concavities and convexities that were manipulated always appeared as features of distinct contours within distinct shapes, with no single exact shape or contour appearing more than once. While this method of random generation ensured a wide range of shapes, it also precluded tight control over the exact geometry of the convexities and concavities being compared. For example, the magnitudes of the turning angles at the convexities and concavities are not precisely controlled using this random-generation technique. Similarly, the randomly generated shapes generally contained more convex vertices than concave vertices—a necessary geometric consequence of the shapes being defined by closed contours. (Closed contours necessarily contain more cumulative positive curvature than negative curvature, because otherwise they would not eventually close on themselves.) In Experiment 1 we minimize this problem by placing the same contour to be changed in both concave and convex conditions, thus equating the geometry to the extent possible. 

Experiment 1 employed a highly controlled stimulus type to test decisively whether the advantage in change sensitivity observed in Experiment 1 is indeed attributable to the sign of curvature. This was accomplished in two ways. First, every change used in Experiment 1 was presented in two different versions, once as a convex change and once as a concave one. That is, shapes in Experiment 1 were generated as pairs, in which the same randomly generated contour belonged to two distinct shapes with opposite sides assigned to the inside of the shape. Using this method, we were able to eliminate any unintended differences that may have contributed to the asymmetry observed in Experiment 1. Frequency of convexities and concavities, the magnitudes of their turning angles, and any other incidental geometric factors along this randomly generated contour were all necessarily equated. 

A second aspect of the design of Experiment 1 addressed the relationship between concavities and convexities within a single shape. Inherent in shape geometry is the fact that concavities must be neighbored by convexities. Thus, making a concavity more or less pronounced through the shifting of a vertex often generates similar alterations to neighboring convex contour (see again Figure 2). One might thus wonder whether the sensitivity advantage demonstrated in Experiment 1 necessarily implies heightened representation of the concave vertices themselves, or whether it might also reflect sensitivity to changes in the neighboring convexities. 

In Experiment 1 we use a more complex method for generating changes, ensuring that all vertices altered by a given shape change have the same sign of curvature (i.e., the vertex being enhanced or diminished and its two immediate neighbors are either all convex or all concave). We accomplished this by creating base shapes so that each vertex to be changed was flanked on each side by at least one additional vertex of the same sign of curvature (Figure 5). When the vertex in question was moved, the two neighboring vertices that were also affected shared the same sign of curvature. Thus, it was impossible for a concave change to affect a convex vertex or vice versa. As a result, any systematic difference in performance obtained between convex and concave vertices would not in any way be attributable to collateral changes to neighboring vertices with opposite sign of curvature. 

Figure 5

 

Each Attneave egg was produced by dividing an ellipse along a single randomly generated jagged boundary, resulting in two separate shapes with a shared boundary. The vertex where the change took place (vertex 3) was flanked by vertices of the same curvature sign (2 and 4).

Method

Observers

A new group of 12 Rutgers undergraduates participated for course credit. 

Stimuli

As in Experiment 1, each stimulus consisted of a base shape and changed versions of that shape. However, unlike Experiment 1, the base shapes were generated in pairs: A randomly generated polygonal contour was used to divide an ellipse (with aspect ratio =.8) into two halves along its major axis, thereby creating two shapes containing the same jagged boundary (see Figure 5). By design, the two shapes in a pair share an identical contour segment, but with opposite sign of curvature at each point of the common contour (e.g., a convex vertex on one shape is a concave vertex on the other shape, as in Attneave’s famous “divided egg“; Attneave, 1971). 

Also, as in Experiment 1, the changed shapes in Experiment 1 were generated by shifting a single vertex of a convexity or concavity. However, as noted above, because the manipulation of a single vertex actually affects three separate angles (the angle at the vertex itself, plus the two neighboring angles), it was important to ensure that the vertices neighboring the shifted vertex were not concave when we were making a convex change. To achieve this, the manipulated vertex in this experiment was always the apex of a pentagonal sequence of vertices, where the three inner angles formed were all of the same sign of curvature (see Figure 5). This constraint, applied both to the base and changed shapes, ensured that convex changes involved only convexities and concave changes only concavities. 

Design and procedure

Based on a pilot study, Experiment 1 used a longer stimulus duration of 500 ms (250 ms longer than in Experiment 1) to allow subjects to perform at above chance levels. This reduced performance was presumably due to the greater complexity of the shapes used in this experiment. Otherwise, the task and procedure were identical to Experiment 1. 

Results

Proportion correct was higher for concave (mean = 63.96%, SE = 2.47%) than convex (mean =56.53%, SE = 2.66%) changes (see Figure 6). Data were converted to d′ for analysis of variance (concave mean =.93, SE =.13; convex mean = .72, SE =.13). The difference between concave and convex change was significant, F(1,11) = 6.35, p < .03. There was no significant effect of change direction (p > .2). 

Figure 6

 

Results of Experiment 1. Error bars indicate standard error.

Discussion

Experiment 1 presents clear evidence for the importance of concavities in shape representation. Because precisely the same contour manipulations were presented as either concave or convex changes—with other geometric variables equated—the observed differential sensitivities can only have been due to this difference in sign of curvature. And because the contours were constrained so that a given change involved exclusively either concavities or convexities—without ever affecting a vertex of the opposite sign—these results provide unambiguous evidence for the special importance of concavities in shape representation. Because of the design of Experiment 1, we can safely conclude that the sensitivity difference cannot be attributed to any unknown confounding differences in contour geometry. 

We note that the magnitude of the effect in Experiment 1 is slightly diminished compared to that in Experiment 1. This is not surprising given that the changes introduced in Experiment 1 were quite a bit more subtle in nature. In particular, because of the pentagonal construction we used in the neighborhood of each change vertex, the changes affected only a small portion of each concavity and convexity (again see Figure 5). Nevertheless, the basic result remained: The sensitivity asymmetry between concave and convex changes in Experiment 1 exactly paralleled that found in Experiment 1. 

General discussion

The recent influx of change blindness phenomena has been taken as evidence regarding the general capacity of visual representation. We have argued that differential “blindness” or sensitivity can also be seen as mapping the relative representational importance of the stimulus properties that the changes are affecting. In other words, asymmetry in sensitivity can be seen as reflecting an underlying representational asymmetry. In the current work, we have applied this logic to investigate the role of sign of curvature in visual shape representation. Our main finding in this study has been that the visual system is more sensitive at boundary segments of negative curvature than at corresponding segments of positive curvature, even when these changes do not alter the number or location of parts. 

The heightened sensitivity to changes in concavities that we observed directly argues for the special role of negative-curvature regions in shape representation. The differential sensitivity was especially impressive in Experiment 1 because, by manipulating figure-ground relationships, we were able to present the very same vertices, embedded in the very same contour segments, as either concavities or as convexities. On the shapes of real object boundaries, there are in fact geometric asymmetries between convexities and concavities (e.g., in their relative frequencies and total curvature) that arise from the mathematical implications of a contour being closed (see Feldman & Singh, 2005). However, by presenting each randomly generated contour twice—once with one side, and then with the other, as the figural shape—we were able to perfectly equate such geometric differences in our stimuli. The results of Experiment 1 thus demonstrate that the asymmetry in sensitivity arises in the way the shape is encoded, and not just as an inevitable consequence of the geometry. 

Importantly, our results also demonstrate that the heightened sensitivity at concavities is not reducible to a sensitivity to changes in qualitative part structure. Given the representational prominence of a part skeleton in shape representation, it is reasonable to suppose that observers would be especially sensitive to changes that alter, for example, the location of partcuts, the topological structure of the axial skeleton, or the number of parts—and indeed they are. (Our previous study found elevated sensitivity to contour changes that induced such gross changes to the part skeleton; Barenholtz et al., 2003; see also Keane et al., 2003.) The current experiments, however, did not involve any such coarse changes: The changes here were relatively small in magnitude, and never induced a change in the sign of curvature at any vertex, effectively guaranteeing that overall part structure was preserved. Nevertheless, we still found that subjects were reliably more able to detect changes within concavities compared to otherwise equivalent changes within convexities. 

At first glance, our results may appear to contradict previous results of Driver and Baylis (1995), which showed that convex segments of previously shown shapes, when presented in isolation, are identified more readily than concave segments. However, our results are in complete accord with, and complement, those of Driver and Baylis. The natural interpretation of Driver and Baylis’ results (and one that they themselves espouse) is that obligatory mechanisms of part decomposition divide shapes at negative minima of curvature—informally, points of sharp concavity—and this decomposition results in roughly convex parts. Because these segmented parts constitute the natural subunits in an object’s representation, they are subsequently identified more easily than are corresponding concave segments (which typically contain fragments from two different parts, and are therefore unnatural as perceptual units of a shape). 

The flip side of this obligatory process of part decomposition is that to achieve the decomposition, negative minima of curvature must first be singled out as candidate segmentation boundaries. A useful analogy here may be the process of edge finding: Although what the visual system ultimately “cares about” is representing objects and surfaces, it initially must devote a great deal of computational resources to finding edges and contours—partly because they provide candidate boundaries between distinct objects. Just as mechanisms of object segmentation require that edges first be identified and highlighted as candidate boundaries between objects, similarly, mechanisms of part segmentation require that regions of sharp concavity first be identified and highlighted as candidate boundaries between parts. And these are exactly the points where, in our experiments, changes were most detectable. Thus, Driver and Baylis’ part-segmentation task and our change-detection task are simply accessing the same fundamental process of part decomposition at negative minima of curvature, but in complementary ways. Whereas Driver and Baylis’ task requires matching a shape fragment to an entire shape, ours requires detecting changes across two presentations of an entire shape. We find heightened sensitivity at the contour regions that define the boundaries between these semi-independent parts, namely the concave corners. The concavities are important representationally not because they are the basic units of shape representation, but because they help to delineate the basic units from each other. 

Various studies (Driver & Baylis, 1995; Hulleman et al., 2000; Humphreys & Müller, 2000; Lamote & Wagemans, 1999; Bertamini, 2001; Barenholtz & Feldman, 2003; Barenholtz et al., 2003) have now shown a specific behavioral preference for either concave or convex contour in specific experimental contexts. Whether the preference attaches to concavities or convexities depends on the experimental task, but as we have suggested, in both cases it simply highlights the importance of the sign of curvature in the visual analysis of shape. This importance is also high-lighted by recent single-cell recordings in area V4 of the monkey cortex, where neurons are found to display differential sensitivity to either convex or to concave extrema of curvature (Pasupathy & Connor, 1999, 2001). As a group, these studies suggest differential processing of shape contour based on sign of curvature. The exact format of the representations served by such processing remains an important topic for further research. 

Acknowledgments

We are grateful to two anonymous reviewers for comments on the manuscript. EHC and MS were funded by National Science Foundation (NSF) Grant 0216944, JF by NSF Grant 9875175 and National Institute of Health (NIH) Grant R01 EY15888, and EB by National Institutes of Health Grant T32-MH19975-03. 

Commercial relationships: none. 

Corresponding author: Elias H. Cohen. Email: . 

Address: Department of Psychology, Rutgers University—New Brunswick, 152 Frelinghuysen Rd., Piscataway, New Jersey, 08854.