Modulation of visually guided action by the image and familiar sizes of real-world objects

Abstract

In daily life, two aspects of real-world object size perception—the image size of an object and its familiar size in the real world—are highly correlated. Thus, whether these two aspects of object size differently affect goal-directed action (e.g., manual pointing) and how have scarcely been examined. Here, participants reached to touch one of two simultaneously presented objects based on either their image or familiar size, which could be congruent or incongruent (e.g., a rubber duck presented as smaller and larger than a boat, respectively). We observed that when pointing to target objects in the incongruent conditions, participants’ movements were slower and were more curved toward the incorrect object compared with the movements in the congruent conditions. By comparing performance in the congruent and incongruent conditions, we concluded that both image size and familiar size influenced action even when task irrelevant, indicating that both are processed automatically (Konkle & Oliva, 2012a). Image size, however, showed influence earlier in the course of movements and more robustly overall than familiar size. We additionally found that greater relative familiar size differences mitigated the impact of image size processing and increased the impact of familiar size processing on pointing movements. Overall, our data suggest that image size and familiar size perception interact both with each other and with visually guided action, but that the relative contributions of each are unequal and vary based on task demands.

Introduction

In 2007, an art installation appeared floating in the harbor of Saint-Nazaire, a small town in Western France. “Rubber Duck” by Florentijn Hofman quickly gained international attention due to the unexpected scene created by a toy that typically measures 5.5 cm in height dwarfing nearby boats at 32 m. The familiar size of a rubber duck, the size that we know it typically would be based on past experience, and the image size of “Rubber Duck,” the size the piece appeared visually to viewers, were dramatically in conflict. 

In normal daily life, the familiar size and image size of objects are highly correlated. When presented in the same context, real-world objects that we know to be relatively small in the real world, such as rubber ducks, typically appear smaller than objects such as boats, which we know to be larger. Even when the sizes of objects on the retina vary, they are integrated with their environment via size constancy mechanisms. Thus, taking size constancy into account, in the real world image size and familiar size are very rarely in conflict. Consequently, image size and familiar size processing are highly confounded—when we see an object and its image size and familiar size are congruent, how each of these two aspects of size affects our perception and goal-directed action is difficult to disambiguate. 

Attempts have recently been made to disentangle image size and familiar size perception. Konkle and Oliva (2012a) implemented a Stroop-like paradigm, in which pairs of two objects were presented at different image sizes on the screen. Participants were asked to indicate which image size was bigger or smaller by key press, while their familiar sizes were task irrelevant. This experiment demonstrated that incongruence between familiar size and image size (e.g., a rubber duck presented with a larger image size than a boat) results in a “familiar size Stroop effect,” captured by slower reaction times for image size judgments. This result suggests that these two highly different aspects of real-world object size, image size and familiar size, are both processed automatically, as in the classic Stroop effect (Stroop, 1935). 

However, despite disambiguating the two to some degree, Konkle and Oliva (2012a) only examined the unidirectional influence of familiar size on image size, as opposed to the bidirectional influence of each on the other. Furthermore, binary categories were used for large and small image sizes, and large and small familiar sizes, although both exist on a continuum in the real world. Therefore, the relative contributions of each type of object size on perception and other visual processes, such as guiding goal-directed actions, bear further investigation. 

Object size is a key component of vision for action in addition to vision for perception, given that size constrains how we interact with the objects in our environment. For example, the width of a target modulates the speed of goal-directed pointing movements to it (Fitts, 1954), and grip aperture scales to the image sizes of objects (Jeannerod, 1984). Prior research has also extensively explored how dissociating the veridical physical and perceived image sizes of objects using contextual size illusions such as the Ebbinghaus, Ponzo, and Müller–Lyer illusions, affects various goal-directed action such as manual pointing and grasping, as well as eye movements (e.g., Bernardis, Knox, & Bruno, 2005; Binsted & Elliott, 1999; de Grave, Franz, & Gegenfurtner, 2006; Franz, 2001; Gamble & Song, 2017; Glover & Dixon, 2002; Knol, Huys, Sarrazin, Spiegler, & Jirsa, 2017; Milner & Goodale, 2008; Van Donkelaar, 1999). 

However, less is known about how familiar size influences action. Image size is a low-level visual feature—it is represented retinally and in V1, one of the earliest visual processing regions (Murray, Boyaci, & Kersten, 2006). In contrast, familiar size is a high-level visual feature requiring object identification and the recruitment of memory, and it is represented in a later visual processing region in the temporoparietal cortex (Konkle & Oliva, 2012b). 

Although conflict between image size and familiar size was demonstrated in the aforementioned paper (Konkle & Oliva, 2012a) using a discrete behavioral response paradigm, it is not known how action would respond to conflict between the two, again given that such scenarios are highly uncommon in the real world. It has been shown that discrete behavioral responses and action measures can produce different results for the same perceptual decision-making task. For example, it is typically thought that strongly salient distractors capture more attention and are more disruptive than weakly salient distractors (Itti & Koch, 2001; Theeuwes, 2010). Counterintuitively, Moher, Anderson, and Song (2015) found dissociable effects of salience on discrete key-press and goal-directed actions such as pointing. In a visually guided pointing task, they required participants to reach to a shape-defined target while trying to ignore salient distractors. They observed that highly salient objects impacted hand movement trajectories less than less salient objects did. Thus, a strongly salient distractor triggers suppression during goal-directed action, resulting in enhanced efficiency and accuracy of target selection relative to when weakly salient distractors are present. In contrast, in a task requiring a key press to select a target, they found greater attentional interference from strongly salient distractors, reflected in slower reaction times. This counterintuitive result suggests that sufficiently strong distractors may trigger suppression, but only when a physical movement is required. These results also underscore the value and necessity of combining visually guided actions with traditional perceptual approaches to fully understand how we resolve competing internal processes to achieve behavioral goals (Moher et al., 2015). 

Furthermore, the continuous nature of manual pointing movements has provided new insights into the temporal evolution of cognitive processes including language processing, numerical cognition, attention allocation, social perception, and cognitive control (for reviews, see Dotan, Pinheiro-Chagas, Al Roumi, & Dehaene, 2019; Erb, 2018; Song, 2017; Song & Nakayama, 2009). Previous studies have demonstrated that the evolution of reach trajectory curvature while selecting a target among alternative choices can reveal various aspects of decision-making, such as timing of information process and the degree of competition (Erb, Moher, Sobel, & Song, 2016; McKinstry, Dale, & Spivey, 2008; Moher & Song, 2019; Song, 2017; Song & Nakayama, 2009). For example, Song and Nakayama (2009) asked participants to determine whether a single Arabic numeral presented in a central square was less than, greater than, or the same as the number 5 by reaching to one of three corresponding squares on the screen. They observed that the greater the numeric deviation between the target and the number 5, the greater the deviation of the trajectory from the standard trajectory. This provides direct evidence that the numeric magnitude of a target is spatially encoded and that the proximity and order of numbers are spatially represented along a hypothesized mental number line. Such a methodology can therefore provide a tool to track how competition between the processing of image size and familiar size evolves and in turn resolves over time. 

Overall, gaps in the literature we sought to address in the present study are the potential influence of familiar size (a higher level visual feature) on action, how it compares to the influence of image size (a lower level visual feature) on action, and whether these interactions change over time. In order to address these questions, we employed a paradigm similar to that of Konkle and Oliva's aforementioned Stroop-like paradigm (Konkle & Oliva, 2012a), with the addition of (a) a familiar size judgment task in which image size was the conflicting task-irrelevant feature, (b) parametrically varied familiar object sizes, and (c) continuous action-based responses as opposed to discrete behavioral responses. In Experiment 1, we tested the hypothesis that the higher level feature familiar size and the lower-level feature image size significantly interfere with each other when the two are incongruent, whereas one type of size is task irrelevant. Furthermore, we sought to investigate the relative strength of interference between familiar and image size processing by measuring temporal aspects (e.g., how quickly movements are initiated) and spatial aspects (e.g., how much reach trajectories are curved toward a wrong choice) of goal-directed pointing movements. In Experiment 2, we evaluated the hypothesis that, if familiar size is treated as a spectrum, varying the magnitudes of relative familiar size differences between pairs objects may lead to corresponding graded effects on action and on the interference between image size and familiar size. 

Overall, we hypothesized bidirectional interference between image size and familiar size but recognized that the influence of each might not be symmetrical. In other words, they might both exert influence on visually guided action but not to the same degree or on the same time scale. Specifically, in accord with classical models of object processing (Collins & Quillian,1969; Jolicoeur, Gluck, & Kosslyn, 1984; Rosch, Mervis, Gray, Johnson, & Boyes-Braem, 1976), if image size is a more intrinsic feature with stronger, more fundamental neural correlates, then it could cause more interference when it is task irrelevant and more strongly resist interference when it is task relevant. In this case, we would expect to see image size exert a relatively strong influence on the familiar size task and familiar size exert a relatively weak influence on the image size task. Furthermore, if there is a temporal component to the asymmetry, we would expect to see image size influence movements earlier in the familiar size task and familiar size influence movements later in the image size task. 

On the other hand, if familiar size is represented more strongly, we would see familiar size exert a relatively strong influence on the image size task and image size exert a relatively weak influence on the familiar size task. Similarly, we could expect to see familiar size influence movements earlier in the image size task and image size influence movements later in the familiar size task. 

Experiment 1: Congruency between target image size and familiar size in visually guided action

Methods

Participants

Fourteen right-handed participants (eight women; mean age, 25.7 years) with normal color vision and normal or corrected-to-normal visual acuity completed both tasks within Experiment 1. Participants provided their informed consent and were compensated monetarily ($10/hour) or with course credit for their participation. The experimental protocol was approved by the Brown University Institutional Review Board in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving humans. 

Apparatus

Stimuli were presented on an upright Plexiglas display facing the seated participant at a distance of approximately 55 cm. A projector behind the display projected a screen measuring 44.1 × 33.0 cm (43.7° × 33.4° visual angle), which participants viewed binocularly. Stimulus presentation was conducted using custom software designed with MATLAB (MathWorks, Natick, MA) and Psychtoolbox (Brainard, 1997). Three-dimensional hand position was recorded with a LIBERTY electromagnetic position and orientation recording system (Polhemus, Colchester, VT) at a rate of 160 Hz with a measuring error of 0.3 mm root mean square. A motion-tracking marker was fastened to the tip of the right index finger of each participant using a Velcro strap. A foam starting block placed 27 cm in front of the participant, between the participant and the display (28 cm from the display), served as the starting position on which the index finger rested at the beginning of each trial. 

Stimuli

Stimuli were adapted from the Konkle and Oliva “object size Stroop” database (Konkle & Oliva, 2012a). All stimuli were presented on a white background. A black eye fixation cross, measuring 7 × 7 mm (0.73° of visual angle) appeared at the center of the screen before each trial. As shown in Figure 1A, two real-world objects were displayed, one to each side of fixation (14.2 cm or 14.7° measured from fixation to target center, 35.5 cm diagonally from the foam starting block). Sixteen unique object pairs were used for the image size task, which required participants to judge the sizes of the objects as they appeared on the screen, and another 16 object pairs for the familiar size task, which required participants to judge the sizes of the objects in the real world. An additional two unique object pairs were used for pre-experiment practice blocks. Thus, 34 total object pairs were used, with each individual object being assigned to only one other object. Example object pairs are shown in Figure 1B. 

Figure 1.

 

Task and stimuli. (A) While the index finger was positioned on the starting block, trials began with the eye-fixation cross for a variable period between 100 and 250 ms. When stimuli were presented, participants had up to 2000 ms to complete the reach-to-touch hand movement indicating their choice. (B) The image size and familiar size of the object could be presented as congruent (e.g., the left image, where the rubber duck has a smaller image size than a boat) or incongruent (e.g., the right image, where the rubber duck has a larger image size than a boat). Additionally, objects were counterbalanced such that the larger and smaller image size and familiar sizes were presented in both left and right positions. (C) Examples of paired objects. The left column shows examples of incongruent conditions, and the right side shows congruent conditions.

In each trial, one object had a larger relative familiar size and the other a smaller relative familiar size (see Figure 1C). Familiar sizes of objects ranged from 3 cm diagonally (a die) to 8776 cm diagonally (a cathedral), as reported by Konkle and Oliva (2012a). In parallel, each trial had one object presented with a larger image size and one presented with a smaller image size. Image sizes, as reported by Konkle and Oliva (2012a), were designed such that the objects with “small” image sizes were bounded by a rectangle with a diagonal equal to 17.5% of our screen height (5.8 cm or 6.0°), and objects with “large” image sizes were bounded by a rectangle with a diagonal equal to 30% of our screen height (9.9 cm or 10.3°). This method was selected to account for the variations in the aspect ratios of real-world objects (Konkle & Oliva, 2011; Konkle & Oliva, 2012a; Kosslyn, 1978). 

In congruent trials, the object with the larger familiar size was presented with a larger image size and the object with the smaller familiar size was presented with a smaller image size. For example, in a congruent trial, a rubber duck would be presented as 5.8 cm diagonally and a boat as 9.9 cm diagonally. In an incongruent trial, relative familiar and image sizes would be incongruent; for example, a rubber duck would be presented as larger on the screen (9.9 cm diagonally) and a boat as smaller on the screen (5.8 cm diagonally). 

Procedure

Two blocks of each task (image or familiar size judgment) were performed in an ABBA order, with image size and familiar size tasks assigned A or B randomly. Task instructions appeared at the beginning of each block (“make your selections based on the sizes you know the objects are in the real world” for the familiar size task, and “make your selections based on the sizes of the objects on the screen” for the image size task). 

Each block was broken into two sequential sub-blocks, each of which contained trials with the same task instructions (“choose the object that is larger” or “choose the object that is smaller”). For example, in an image size task block, for the first sub-block participants would select the object with the larger image size; in the second sub-block, the object with the smaller image size. The order of these sub-blocks was randomly determined (i.e., larger–smaller or smaller–larger) and reversed in the second block of each task. In total, each of the 32 experimental object pairs was repeated eight times for a total of 128 trials per experimental task. 

At the beginning of each participant's session, a nine-point calibration was conducted for the tracker. Participants also completed a practice block of 16 trials of the relevant size judgment task before the first experimental block of that task. In both the image size and familiar size tasks, when participants rested their index finger on the starting position, each trial began with an initial fixation-cross presented for a variable amount of time (100–250 ms). Participants were asked to hold their eyes on the fixation cross, although it was not monitored via eye tracker. This was followed by the presentation of the object pair. Participants were instructed to make their decisions as quickly and accurately as possible by reaching out and touching the selected object on the screen. In both tasks, the display was presented for a maximum of 2000 ms. Auditory feedback was given when participants touched the display indicating either correct object selection (single high beep) or incorrect object selection (single low beep), or if the time limit had been exceeded (double low beep). An example trial of the task is diagrammed in Figure 1B. 

Data analysis

Data analysis procedures were largely adapted from methods reported in our previous work (Gamble & Song, 2017; Moher & Song, 2013, 2014). Using custom MATLAB software, we conducted offline data analysis on the pointing movement data. Movement velocity was calculated from the three-dimensional position traces after filtering with a low-pass filter (cutoff frequency of 10 Hz). An algorithm using velocity criteria of 10 cm/s detected the beginning and end of pointing movements. The identification of these movements by the algorithm was visually inspected to verify its accuracy for every trial (Gamble & Song, 2017; Moher et al., 2015; Moher & Song, 2013, 2014); for trials in which the default threshold clearly missed capturing part of the movement or included substantial post-selection movement, thresholds were adjusted manually to more appropriate levels for that trial (∼ 1% of all trials). Pointing movements were classified as correct responses if they landed within a standardized target boundary used for all targets (6 × 6 cm or 6.2° × 6.2° of visual angle). Thus, accuracy was defined as the percentage of correct responses. 

Initiation latency (IL) was defined as the time elapsed between stimulus onset and pointing movement onset. Movement time (MT) was defined as the time elapsed between movement onset and movement offset/target landing. Maximum curvature was calculated by tracing the path of the hand and calculating an ideal direct path between the starting and end points of the movement. The perpendicular deviation of the hand position from the ideal path was calculated at every time point over the course of the movement. The maximum of these perpendicular deviation lengths divided by the length of the ideal path results in a unitless ratio, referred to as maximum curvature. Larger ratios thus represent a greater maximum deviation from the ideal path and greater overall curvature in the movement's trajectory. In order to compare congruent and incongruent trials within subjects for each measure (IL, MT, and maximum curvature) and to compare the image size and familiar size tasks directly, we performed 2 × 2 repeated-measures analyses of variance (ANOVAs) with factors of congruency (congruent vs. incongruent) and task (image size vs. familiar size judgments) for each measure. We also report partial eta squared, with values of 0.2 indicating a small effect size, 0.5 indicating a medium effect size, and 0.8 indicating a large effect size (Cohen, 1973). Within each task (image size and familiar size), we additionally performed paired t-tests comparing congruent and incongruent trials within subjects for each measure (accuracy, IL, MT, and maximum curvature). 

In addition to these analyses, we also examined the evolution of the reach trajectories over the course of the hand movements to the screen. In order to average and compare across participants and across trials, which naturally vary in length, we normalized all hand movements. First, movements to the left and right sides of the screen were collapsed. Additionally, we normalized all movements for each participant to 101 evenly spaced data points based on the linear distance of the hand from its starting point to the screen (the z-dimension, or “forward” from the participant). Then, to directly compare performance between the congruent and incongruent trials in each task, we focused on the lateralized horizontal movement on the x-dimension (left–right direction), the dimension along which the two competing potential targets differ. Specifically, at each of the 101 points, we calculated a difference between the averaged trajectories on the x-dimension between the congruent and incongruent conditions (x-pos_incongruent – x-pos_congruent). The resultant difference score was calculated as positive if there was a measurable difference between the average position for the congruent and the incongruent conditions, indicating attraction of the hand movement toward the incorrect response alternative and significant interference from the task-irrelevant feature. 

Comparing this difference score to zero revealed the points at which there was a significant difference in position between congruent and incongruent trials. Significant differences in position indicate interference from the task-irrelevant feature (e.g., in the familiar size task, difference scores significantly above zero indicate significant interference from image size). 

These normalized positions were analyzed using a cluster-based permutation test. In order to correct for the multiple comparisons arising from conducting 101 t-tests, we used the Monte Carlo method to sample the datapoints in 500 iterations and find those points that were more significant (at a 95% confidence level) than the calculated test statistic. This analysis was performed using the FieldTrip toolbox for MATLAB (Oostenveld, Fries, Maris, & Schoffelen, 2011). The key data point in this subset is the first datapoint that was significantly greater than zero, indicating the point on average at which the trajectories of congruent and incongruent trials begin to deviate. 

Results

We excluded 5.34% (±1.09% SE) of trials from data analysis for the image size task and 5.19% (±1.48%) for the familiar size task due to technical issues (e.g., expected occasional sampling drop). Accuracy was 97.47% (±2.79%) for the image size task and 97.17% (±2.83%) for the familiar size task. All subsequent analyses were restricted to correct trials. 

Image size task

As seen in Figure 2A, in the image size task (leftmost bars) participants performed more accurately on congruent trials (black bars) compared to incongruent trials (cyan bars). This result indicates that interference from familiar size was strong enough to lead to incorrect choices when familiar size was in conflict with image size. This effect was supported by an overall main effect of congruency, F(1, 13) = 5.35, p = 0.038, η_p² = 0.292, with no interaction between congruence and task, F(1, 13) = 0.71, p = 0.794, η_p² = 0.005. Further, a post hoc t-test between congruent and incongruent trials for the image size task similarly showed a significant effect, t(13) = 2.83, p = 0.014, d = 0.697. 

Figure 2.

 

Results for congruent versus incongruent trials in the image size and familiar size tasks in Experiment 1. All error bars represent between-subject standard error. (A) Accuracy was lower for the incongruent conditions than the congruent conditions in both tasks. (B) Initiation latency was faster for congruent compared to incongruent trials in both tasks and was faster overall for the image size task. (C) Movement time was faster for the congruent conditions in both tasks and was faster overall for the image size task. (D) Movements were more curved in the incongruent trials for both tasks. Note that curvature values represent a unitless ratio.

A congruency effect was also seen in initiation latency (Figure 2B) such that participants were faster overall to initiate movements when image size and familiar size were congruent compared with when they were incongruent, F(1, 13) = 28.02, p < 0.001, η_p² = 0.683, with no interaction between congruence and task, F(1, 13) = 0.88, p = 0.365, η_p² = 0.063; t(13) = –2.84, p = 0.014, d = –0.199. Similarly, congruency also affected online movement time (Figure 2C) such that movements were executed more quickly overall when image size and familiar size were congruent compared with incongruent, F(1, 13) = 41.14, p < 0.001, η_p² = 0.760, with no interaction between congruence and task, F(1, 13) = 0.519, p = 0.484, η_p² = 0.038; t(13) = –3.62, p < 0.001, d = –0.259. 

Finally, we examined maximum curvature (Figure 2D) over the course of the movement as a measure of how much movements were pulled toward the incorrect object before ultimately landing on the correct object. Figure 2D shows that the average maximum curvature of pointing movements was greater when image size and familiar size were incongruent compared when they were congruent, showing that familiar size interfered with the image size task (e.g., when the task was to choose the larger object based on its image size, participants’ hand movements were drawn toward the object with the larger familiar size), F(1, 13) = 43.24, p < 0.001, η_p² = 0.760, with no interaction between congruence and task, F(1, 13) = 0.79, p = 0.391, η_p² = 0.057; t(13) = –5.24, p < 0.001, d = –1.18. 

Overall, the results from the image size task indicate that the higher level feature, familiar size, significantly interferes with the lower level feature, image size, when the two are incongruent, even when familiar size is task irrelevant. This is consistent with prior findings regarding the familiar size Stroop task (Konkle & Oliva, 2012a). 

Familiar size task

Figure 2A shows that, as in the image size task, participants were more accurate on average for congruent compared to incongruent trials for the familiar size task (rightmost bars). This demonstrates that image size interferes with familiar size even when it is task irrelevant, suggesting a bidirectional influence of both aspects of size on reaching. This effect was supported by the aforementioned main effect of congruency overall, F(1, 13) = 5.35, p = 0.038, η_p² = 0.292, with no interaction between congruence and task, and a post hoc t-test between congruent and incongruent trials for the familiar size task, t(13) = 2.43, p = 0.030, d = 0.802. Similarly, there was also an effect of congruency in IL (Figure 2B), F(1, 13) = 28.02, p < 0.001, η_p² = 0.683, with no interaction between congruence and task, t(13) = –3.58, p = 0.003, d = –0.328, and MT (Figure 2C), F(1, 13) = 41.14, p < 0.001, η_p² = 0.760, with no interaction between congruence and task, t(13) = –2.95, p = 0.011, d = –0.461, such that participants were slower when image size and familiar size were incongruent. Finally maximum movement curvature (Figure 2D) was significantly greater when image size and familiar size were incongruent, as was also the case in the image size task, F(1, 13) = 43.24, p < 0.001, η_p² = 0.760, with no interaction between congruence and task, t(13) = –3.91, p = 0.002, d = –0.876. 

Overall, these results suggest that there is significant interference from image size in the familiar size task such that performance is impaired when image size and familiar size are incongruent, despite the fact that image size here is task irrelevant. 

Comparing image size and familiar size

We additionally compared performance on the image size and familiar size tasks in all of the aforementioned measures. We observed no difference between the tasks for accuracy, F(1, 13) = 0.068, p = 0.799, η_p² = 0.005, with no interaction between congruency and task, F(1, 13) = 0.071, p = 0.794, η_p² = 0.005, or for maximum curvature, F(1, 13) = 0.636, p = 0.439, η_p² = 0.047, with no interaction between congruency and task, F(1, 13) = 0.789, p = 0.391, η_p² = 0.057. For initiation latency, we observed that participants were faster to initiate movements in the image size judgment task compared with the familiar size judgment task, F(1, 13) = 69.22, p < 0.001, η_p² = 0.842, with no interaction between congruency and task, F(1, 13) = 0.880, p = 0.365, η_p² = 0.063. Similarly, online movement execution was faster, with the faster MTs observed for the image size task, F(1, 13) = 11.96, p = 0.004, η_p² = 0.479, with no interaction between congruency and task, F(1, 13) = 0.519, p = 0.484, η_p² = 0.038. 

Overall, then, participants took longer to plan and execute their movements in the familiar size task compared to the image size task. This result is perhaps related to the fact that familiar size is characterized as high level and image size is characterized as low level, meaning that evaluating familiar size requires more cognitive processing than evaluating image size. For example, according to classical models of object processing, when the visual system has extracted feature information, such as image size, curvature, and depth, basic-level object recognition precedes accessing knowledge about that object, such as its familiar size (Collins & Quillian,1969; Jolicoeur et al., 1984; Rosch et al., 1976). This effect, though, was not seen in accuracy or maximum curvature, suggesting that the key difference between the two tasks is temporal in nature. 

Time course of movement modulation in the image size and familiar size tasks

Our analysis of hand movement trajectory revealed key differences in maximum curvature between congruent and incongruent trials, in both the image size and familiar size tasks (Figure 2D). In addition to the spatial measure of maximum curvature, which refers to a discrete point in a hand movement, we can additionally examine the impact of incongruency over the course of our continuous movement data. Comparing hand positions over the course of the movement for congruent and incongruent trials in the image size and familiar size tasks provides insight into when image size and familiar size interact, in addition to previous measures of how much they interact. As mentioned previously, movements were normalized to space with respect to reach distance from the starting point to the target (see Reach trajectory normalization in Methods for details). Space-based normalization has the advantage of minimizing a potential confound from MT differences across trials. Thus, in evaluating the evolution of conflict between image size and familiar size, we discuss the percentage into the course of the movement at which differences occur (Gallivan & Chapman, 2014). 

Figure 3 shows average normalized hand movements in real space for the image size task where familiar size is the interfering feature (Figure 3A) and the familiar size task where image size is the interfering feature (Figure 3B). For both tasks, trials where image size and familiar size were congruent are shown in black, and trials where image size and familiar size were incongruent are shown in cyan. Although the trials were counterbalanced such that reaches were performed with equal frequency to the left and right sides of the screen, here we have collapsed all trials along the midline such that greater values on the x-axis represent hand positions closer to the correct target object, and smaller values on the x-axis represent hand positions closer to both the starting position and the incorrect non-target object. In both tasks, we observed that the movements were relatively direct to the target in the congruent trials (black lines). In contrast, movements were more pulled by the task-irrelevant feature and therefore curved toward the distractor object in the incongruent trials (cyan lines), leading to significant deviations between movement paths for congruent and incongruent trials in both tasks. 

Figure 3.

 

Average normalized trajectory of hand movements in congruent (black line) and incongruent (cyan line) trials over the course of accurate reach-to-touch movements across all participants. (A) Image size task trials, where familiar size was the task-irrelevant interfering feature. (B) Familiar size task trials, where image size was the task-irrelevant interfering feature. Movements to the left and right targets are collapsed across the midline, such that positions farther to the right are closer to the target, and those farther to the left are closer to the midline and the incorrect object. Ribbons represent 95% confidence intervals. Hand and target are not to scale. (C) Difference scores between congruent and incongruent trials representing the magnitude of interference by incongruency in the image size and familiar size tasks. Greater positive values represent greater attraction to the alternative (incorrect) object in incongruent trials (i.e., strength of interference from the task-irrelevant feature). Scores in the familiar size task (where interference was from image size) rise significantly above zero earlier and peak earlier than the scores for the image size task (where interference was from familiar size). The ribbons represent 95% confidence intervals.

In order to analyze the deviation between congruent and incongruent trials over time, and particularly to compare the deviations in the image size with the deviations in the familiar size tasks over time, we calculated a difference score between normalized congruent and incongruent trials along the x-dimension in each task, as shown in Figure 3C. Our goal in calculating a difference score was to isolate and quantify the interference from the task-irrelevant feature. Given that the correct target (defined by the task-relevant feature) and the incorrect distractor object (defined by the task-irrelevant feature) were separated in space on the screen only in the x-dimension, we analyzed this dimension exclusively to most precisely examine this interference. 

Here, the difference between incongruent and congruent trials is depicted in red for the image size task and in blue for the familiar size task. In the familiar size task (blue solid line), where image size was the competing feature, difference scores are significantly greater than zero between 15% and 91% (two vertical blue dotted lines) of the movement, indicating that the average paths of the congruent and incongruent trials were significantly different in this span. In contrast, in the image size task (red solid line), where familiar size was the competing feature, the congruent and incongruent trajectories deviated later than in the familiar size task, approximately between 36% and 90% into the movement (two vertical red dotted lines). Thus, congruency becomes a factor in the familiar size task after a smaller portion of the movement, or, put another way, image size influences movements at an earlier proportion than familiar size does (Bennett, 2007). There is no significant difference in the points at which the difference scores return to zero, indicating that congruent and incongruent trials converge at similar points of the movement for the image size and familiar size tasks. 

Taken together, these results suggest that the image sizes of objects impact movement to a greater degree than familiar size does, as demonstrated by a greater effect of incongruency in the familiar size compared to the image size task. Not only is the magnitude greater, but the impact of image size is observed earlier in the course of the decision-making process than familiar size is. This suggests that image size may be processed more robustly overall than familiar size, although once again both are automatic and robust enough to interfere with the other. 

Summary

The goal of Experiment 1 was to examine if, and how, the processing of image size and familiar size influences goal-directed pointing movements when one type of object size is task irrelevant. Overall, we demonstrated that both the image size and familiar sizes of real-world objects play a role in planning and generating goal-directed action. Specifically, when we manipulated the congruency of the image size and familiar size of a target, each of these two aspects of object perception interfered with the other. This suggests that both image size and familiar real-world size are aspects of object perception and identification that occur automatically, even when task irrelevant, and furthermore are not independent from action. Despite the bidirectional interference, however, image size may be more robustly processed and represented and may interfere with familiar size judgments more than familiar size interferes with it (e.g., Figure 3C). This observation was investigated in more detail in Experiment 2, along with questions regarding absolute and relative familiar size differences and their relationship with action. 

Experiment 2: Effects of familiar size difference magnitude on action

In Experiment 1, like in much of the existing literature, we categorized the familiar size of the two objects presented concurrently by which of the two was “larger” and which was “smaller,” while largely ignoring the magnitude of this relative size difference (Chao & Martin, 2000; Downing, Chan, Peelen, Dodds, & Kanwisher, 2006; Kanwisher, 2001; Macuga & Papailiou, 2012; Wang & MacKenzie, 1999). Thus, it is still unknown whether it only matters that the familiar size of one object is larger than the other, or whether how much larger it is has an impact, as well. For example, does comparing two objects where one is only slightly larger than the other (e.g., a peanut and a paperclip) have the same effect as comparing two objects that are vastly different sizes (e.g., a bathtub and a paperclip)? In Experiment 2, to examine whether differing degrees of real-world size differences impact goal-directed pointing, we expanded our set of real-world objects to include a wider range of familiar sizes and systematically manipulated the magnitude of the difference in familiar sizes between the paired targets. We expected that if familiar size is treated as a graded spectrum, varying the magnitudes of familiar size differences may lead to corresponding graded effects on action and on the interference between image size and familiar size. 

Methods

Participants

Twelve new right-handed participants (10 females; mean age, 21.6 years) who did not participate in Experiment 1 and who had normal color vision and normal or corrected-to-normal visual acuity completed both visually guided pointing tasks: familiar size judgment and image size judgment. Of these, 11 (10 females; mean age, 21.7 years) additionally performed a familiar size rating task to validate our relative familiar size category manipulation. Participants provided their informed consent and were compensated monetarily ($10/hour) or with course credit for their participation. The experimental protocol was approved by the Brown University Institutional Review Board in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving humans. 

Apparatus

The same apparatus was used as in Experiment 1. 

Stimuli

Experiment 2 stimuli were similar to those used in Experiment 1. We created image pairs from individual objects in the Konkle and Oliva “object size range” database (Konkle & Oliva, 2011). Images in this database are divided into eight groups based on their familiar real-world sizes. These groups were independently defined and validated by participants (Konkle & Oliva, 2011). Using these eight familiar size groups, we defined four categories of paired images ranging from small familiar size differences to large familiar size differences. Objects in category 1 were selected from within the same familiar size group (e.g., a peanut and a paperclip [both group 1] or a space shuttle and an airplane [both group 8]). Category 2 objects were two groups apart (e.g., a wineglass [group 3] and a teabag [group 1]), category 3 objects were four groups apart (e.g., a cooler [group 5] and a die [group 1]), and category 4 objects were six groups apart (e.g., a car [group 7] and a key [group 1]). All eight groups defined by familiar size by Konkle and Oliva (2011) were represented in all of our four categories defined by relative familiar size difference between pairs. Thus, in this four Category by eight group manipulation, there was a total of 32 image pairs. Examples of pairs from each category, in their congruent and incongruent configurations, are shown in Figure 4A. All image pairs can be seen in Figure 4B. 

Figure 4.

 

Representative examples of familiar size categories. (A) Representative examples of each of the four relative familiar size categories for object pairs in both congruent and incongruent conformations. Categories 1 to 4 have progressively greater differences in familiar size magnitude. (B) Every image pair used in its congruent conformation is arranged by category and group.

Procedure

The procedure for Experiment 2 was the same as in Experiment 1. Our four relative familiar size category manipulation based on the established image dataset was validated by participants’ ratings of how different the familiar sizes of the objects in each pair were on a scale from 0 to 10, representing no difference to extreme size differences. All image pairs from the pointing tasks were presented with a rating bar at the bottom of the screen representing a continuum from 0 to 10. Participants were instructed to use the mouse to click anywhere within the bar to rate how large the relative familiar size difference was for the object pair shown (e.g., “How different are the sizes of a rubber duck and a boat in the real world?”). As in Experiment 1, object pairs were presented with the relative image size and familiar size either congruent or incongruent. This task took place after both the image size and familiar size judgment tasks in order not to bias performance. 

Data analysis

All data analysis was identical to that for Experiment 1, with the following exceptions. The measure rating was defined as the average value given to each object pair when participants were instructed to rate the pairs based on their relative familiar size differences. To analyze the effects of task and congruency in movement accuracy, initiation latency, movement time, and maximum curvature and to compare between the two tasks as in Experiment 1, we performed a series of 2 × 4 × 2 ANOVAs with factors of congruency (congruent or incongruent), category (1–4), and task (image size vs. familiar size judgment). 

Results

In total, 10.75% (±1.02% SE) of trials were excluded from data analysis from the image size task and 10.93% (±1.27%) from the familiar size task due to technical issues (e.g., sampling drop). Accuracy was 89.25% (±1.02%) for the image size task and 88.09% (±1.27%) for the familiar size task. All data analyses were restricted to correct trials. 

Ratings

To validate the four categories that we designed based on relative familiar size difference magnitude, we asked participants to rate this difference for each pair of objects. Additionally, we presented each pair in both its congruent and incongruent conditions, in order to evaluate the effect of congruency on subjective ratings of familiar size difference. 

Figure 5 shows the average ratings assigned by participants to the image pairs in each of the four relative familiar size difference categories (1–4) when the relative familiar and image sizes were congruent (black) and incongruent (cyan). Participant ratings validated our manipulation of creating image pairs that fit these four categories; Figure 6 shows a clear trend of average rating increasing in a stepwise manner across categories 1 to 4 (small to large familiar size difference). This was supported by a 2 × 4 repeated-measures ANOVA with factors of congruency (congruent vs. incongruent) and category (1–4), which revealed a main effect of category, F(3, 30) = 55.80, p < 0.001, η_p² = 0.848. 

Figure 5.

 

Average ratings of relative familiar size differences for image pairs in each relative image size category, when the image size and familiar size of the paired objects are congruent and incongruent. Ratings supported our category definitions, with higher ratings given to categories designed to show larger familiar size differences. There was no significant difference in rating based on congruency. All error bars represent between-subject standard error.

Figure 6.

 

Results for congruent versus incongruent trials across relative familiar size conditions in the image size task of Experiment 2. All error bars represent between-subject standard error. (A) Accuracy was consistent across all conditions. (B) Initiation latency was consistent across all conditions. (C) Movement time was slower overall for categories with larger familiar size differences. (D) Movements were more curved in the incongruent trials overall and for categories with lower relative familiar size differences.

Figure 5 appears to show a trend such that ratings of the familiar size differences were higher on average in the congruent compared to incongruent condition. Such an effect would suggest that participants attended to the image sizes of the objects despite task irrelevance and that congruent image size differences magnified the perceived difference in familiar size. However, there was no significant effect of congruency, F(1, 10) = 0.874, p = 0.372, η_p² = 0.080, and no interaction between congruency and category, F(3, 30) = 0.705, p = 0.557, η_p² = 0.066. This could be due to the absence of a meaningful effect or insufficient power. This latter possibility may reflect the relative weakness of a perceptual button-press task compared with the more robust action measures employed in the main task (which show consistent effects of congruency) in accord with past research (Finkbeiner, Song, Nakayama, & Caramazza, 2008). 

Image size task

Accuracy: As shown in Figure 6A, there were no clear trends in accuracy across conditions in the image size task. There was no main effect of congruency (congruent vs. incongruent), F(1, 11) = 0.223, p = 0.648, η_p² = 0.024, or category (1–4), F(3, 33) = 1.42, p = 0.259, η_p² = 0.136. There was, however, a significant interaction between the two such that the strength of the congruency effect was different in different categories, F(3, 33) = 6.62, p = 0.002, η_p² = 0.424. This difference does not appear to follow a systematic trend across categories as familiar size difference magnitude varies; thus, it is impossible to make any claims about a meaningful effect of category on congruency or vice versa. 

Initiation latency: In order to examine how congruency and the relative differences between the familiar sizes of real-world objects affected movement preparation we again analyzed IL. Figure 6B shows that for the image size task, unlike in Experiment 1, IL was consistent across conditions. There was no significant difference based on congruency, F(1, 11) = 1.27, p = 0.282, η_p² = 0.096, based on category, F(3, 33) = 2.37, p = 0.087, η_p² = 0.165, and no interaction between the two, F(3, 33) = 2.34, p < 0.085, η_p² = 0.166. 

Movement time: Again, to examine how congruency and familiar size difference affected the online control of pointing, we analyzed MT. Figure 6C shows a marginal effect of congruency on MT in the image size task, F(1, 12) = 4.02, p = 0.068, η_p² = 0.251. Additionally we observed that with larger familiar size differences, MTs were slower overall, F(3, 33) = 10.10, p < 0.001, η_p² = 0.457. As in the accuracy data, there was an interaction between congruency and category, F(3, 33) = 5.36, p = 0.004, η_p² = 0.309. Again, though, this effect does not appear to vary systematically with category; Figure 6C shows that categories 2 and 3 displayed greater differences between congruent and incongruent trials than 1 and 4, so we can draw no conclusions about the interaction in terms of MT. 

Maximum curvature: In the image size task, the maximum curvature results are consistent with those seen in movement time and replicate the congruency effect seen in maximum curvature in Experiment 1 (Figure 2D, left). Maximum curvature was greater overall in the incongruent conditions, F(1, 11) = 6.72, p = 0.024, η_p² = 0.359, suggesting interference from familiar size when the two were incongruent (Figure 6D). Maximum curvature was also larger when familiar size differences were larger, F(3, 33) = 7.68, p < 0.001, η_p² = 0.390. There was additionally an interaction between congruency and category, F(3, 33) = 2.99, p = 0.044, η_p² = 0.199, indicating that congruency does not have the same impact across categories where familiar size differences vary. 

Familiar size task

Accuracy: In the familiar size task, we observed several effects in accuracy not seen in the image size task. As seen in Figure 7A, there was a significant effect of congruency such that accuracy was lower when image size and familiar size were incongruent, F(1, 11) = 11.37, p = 0.008, η_p² = 0.558. There was additionally an effect of category such that accuracy was lower for the conditions with smaller familiar size differences, F(3, 33) = 32.48, p < 0.001, η_p² = 0.783. This suggests that judgments of relative familiar size were more difficult when paired objects were closer in their familiar size. Here, familiar size was the task-relevant feature, whereas it was task irrelevant in the image size task. Thus, it is not surprising to see an effect of category here but not in the image size task (Figure 6A). In Figure 7A there appears to be a trend such that the congruency had a smaller effect in larger categories, suggesting that larger familiar size differences were processed more robustly and more successfully resisted interference from image size; however, this interaction was only marginally significant, F(3, 33) = 2.61, p = 0.072, η_p² = 0.225. 

Figure 7.

 

Results for congruent versus incongruent trials across relative familiar size conditions in the familiar size task of Experiment 2. All error bars represent between-subject standard error. (A) Accuracy was higher for congruent compared to incongruent conditions and for larger relative familiar size category. (B) Initiation latency was faster for congruent trials. (C) Movement time was faster for congruent trials and for larger relative familiar size difference categories. (D) Movements were more curved in the incongruent trials overall and for categories with lower relative familiar size differences.

Initiation latency: Figure 7B shows that, unlike in the image size task, ILs were marginally slower for incongruent compared to congruent trials, F(1, 11) = 4.53, p = 0.055, η_p² = 0.274. Additionally, there was a significant effect of category, F(3, 33) = 5.35, P = 0.004, η_p² = 0.308, and interaction between congruency and category, F(3, 33) = 3.48, P = 0.026, η_p² = 0.225. In the image size task, familiar size was the task-irrelevant interfering factor, whereas here in the familiar size task image size is the interfering factor. Overall, familiar size failed to exert any influence on IL in the image size task (Figure 7B), but image size significantly interfered with the speed of movement initiation for the familiar size task (Figure 7B). Additionally, unlike the image size task, differences in familiar size impacted IL, as might be expected given that image size here was the task-relevant feature. However, these differences among categories do not appear to follow any systematic trend but rather seem to be driven primarily by category 3. Thus, no strong claims can be made based on this effect. 

Movement time: Figure 7C shows that, in the familiar size task, movements were faster overall when image size and familiar size were congruent, F(1, 11) = 12.74, p = 0.004, η_p² = 0.515. Additionally, MTs were significantly faster when familiar size differences were larger (categories 1–4), F(3, 33) = 78.06, p < 0.001, η_p² = 0.867. There was no interaction between these factors, F(3, 33) = 0.216, p = 0.885, η_p² = 0.018. As with IL, it is not surprising that category, which is based on relative familiar size difference, would modulate the familiar size task more than the image size task. That said, a clear asymmetry exists between the influence of image size on familiar size and the influence of familiar size on image size, as seen in the difference between congruency effects in Figures 6C and 7C. 

Maximum curvature: As in the image size task, maximum curvature was again greater overall in the incongruent compared to congruent conditions, F(1, 11) = 18.47, p < 0.001, η_p² = 0.606 (Figure 7D). Here, though, there was less maximum curvature overall when familiar size differences were smaller compared to larger (categories 1–4), F(3, 33) = 68.95, p < 0.001, η_p² = 0.852. Again, familiar size here is the task-relevant factor, whereas it is the interfering factor in the image size task, leading to opposite effects of increasing familiar size difference in the two tasks. What is consistent across both is that, with larger familiar size differences, there is a larger effect of familiar size. Similarly, there was an interaction between these two factors such that as relative familiar size difference (category) increased, the incongruency effect decreased, F(3, 33) = 40.96, p < 0.001, η_p² = 0.773. Again, this reflects a decreasing degree of interference from image size as relative familiar size differences increase and is expectedly the reverse of the effect seen in image size. 

Comparisons between image size and familiar size tasks

In addition to the above within-task measures, we again compared the image size and familiar size tasks to each other. We found greater accuracy, F(1, 11) = 6.48, p = 0.031, η_p² = 0.419; faster initiation latency, F(1, 11) = 40.06, p < 0.001, η_p² = 0.770; faster movement time, F(1, 11) = 46.41, p < 0.001, η_p² = 0.795; and smaller maximum curvature, F(1, 11) = 107.25, p < 0.001, η_p² = 0.899, in the image size task than the familiar size task. Taken together, this suggests that the familiar size task may have been more difficult than the image size task, consistent with participants’ subjective reports. 

There were additionally task by congruency interactions in accuracy, F(1, 11) = 7.20, p = 0.025, η_p² = 0.444; movement time, F(1, 11) = 84.62, p < 0.001, η_p² = 0.876; and maximum curvature, F(1, 11) = 10.81, p = 0.006, η_p² = 0.474, and a marginally significant effect in initiation latency, F(1, 11) = 4.37, p = 0.059, η_p² = 0.267. Together, these all point to image size interfering in the familiar size task more than familiar size did in the image size task. 

We also observed interactions between category (i.e., the degree of familiar size difference between paired objects) and task for both movement time and maximum curvature, such that categories with larger familiar size differences led to shorter movement times, F(3, 33) = 84.32, p < 0.001, η_p² = 0.89, and less curvature, F(3, 33) = 85.12, p < 0.001, η_p² = 0.89, in the familiar size compared with the image size task. This effect is not surprising given that familiar size was task relevant in the familiar size task and task irrelevant in the image size task. 

Finally, for maximum curvature, there was a three-way interaction among task, congruency, and category, F(3, 33) = 11.21, p = 0.007, η_p² = 0.51. Such an interaction is expected given both the individual main effects seen in the previous analysis and the way our categories are defined based on magnitude of familiar size difference. In the image size task (Figure 6D), larger familiar size differences (category) led to more curved trajectories, F(3, 33) = 7.68, p < 0.001, η_p² = 0.390. However this effect seems to be driven entirely by the incongruent conditions, as maximum curvature increases across the incongruent conditions based on category, whereas the congruent trials are stable across category, consistent with a category by congruency interaction, F(3, 33) = 2.99, p = 0.044, η_p² = 0.199. This suggests that increasing the familiar size difference increases the interference of familiar size in performing the image size task. However, when familiar size and image size are congruent, familiar size difference plays no role, or a negligible role compared to the perception of image size. 

In the familiar size task, there was an overall decrease in maximum curvature as the familiar size difference (category) increased (Figure 7D), F(3, 33) = 68.95, p < 0.001, η_p² = 0.852. This is the reverse of the effect seen in the image size task because here participants’ decisions were made based on familiar size as opposed to image size, whereas category was defined by relative familiar size difference in both cases. Thus, the lower maximum curvature for the greater familiar size differences may reflect lower task difficulty. Similarly, the interaction effect suggests that the effect of congruency is different across categories. Specifically, the congruency effect is greater in the smaller familiar size difference categories, showing that image size interferes with participants’ ability to judge familiar size more when the familiar size difference is smaller (i.e., the familiar sizes of the paired objects are closer). This decrease in the effect of congruency across category was supported by a post hoc linear trend analysis in which we found that difference scores (curvature_incongruent – curvature_congruent) significantly decreased across category, as familiar size difference increased, r = –0.32, p = 0.028. 

Taken together with converging evidence that image size is processed more robustly than familiar size, this suggests that image size is the greater influence on trajectory overall but that familiar size exerts a larger influence the larger the magnitude of familiar size differences. This increased modulation of movement by familiar size leads to both greater interference from familiar size in the image size task and greater resistance against interference from image size in the familiar size task. 

Time course of movement modulation in the image size and familiar size tasks

As in Experiment 1, we again calculated difference scores from the normalized congruent and incongruent trials in each task to examine the influence of image size and familiar size over time (Figure 8). Specifically, we were interested in the effect of varying the magnitude of the familiar size difference between paired images on the time course of the impact on movement of both image size and familiar size. 

Figure 8.

 

Difference scores between congruent and incongruent trials representing the magnitude of interference by incongruency in the image size and familiar size tasks. Scores in the familiar size task (blue), where image size was the interferer, rise significantly above zero earlier than the scores for the image size task (red), where familiar size was the interferer. The ribbons represent 95% confidence intervals.

We first performed the same time course analysis as in Experiment 1 by comparing the difference scores between congruent and incongruent trials for the image size task (solid red line) and familiar size task (solid blue line). Figure 8 shows that the difference scores were significantly above zero: 53% to 71% into the movement for the image size task (vertical red dotted lines) and 14% to 97% into the movement for the familiar size task (vertical blue dotted lines). This replicates our Experiment 1 findings that image size came online and influenced movements earlier in the course of movements than familiar size did. 

It is worth noting that the precise timing and the magnitude of interference in Experiment 2 did not completely replicate the results in Experiment 1. In the image size task specifically, familiar size interfered later and less robustly in Experiment 2 than in Experiment 1. This could reflect the fact that the range of familiar sizes was much larger in Experiment 2 compared with Experiment 1; thus, there was a wider range of interference effects and more noise overall. 

We additionally compared the influences of image size and familiar size on the movements among categories. Figure 9 shows the time course of difference scores for each category, with the image size task presented in red and the familiar size task in blue. Figures 9A to 9D represent data from category 1 to category 4, which contain the smallest to the largest familiar size difference magnitudes, respectively. 

Figure 9.

 

Time course of different scores across categories in Experiment 2. Red and blue lines represent mean difference scores in the image size task and familiar size task, respectively. The ribbons represent 95% confidence intervals. (A) Category 1, (B) category 2, (C) category 3, (D) category 4.

For the image size task (red), where familiar size was the interfering factor, there was an overall main effect of congruency. However, only category 3 independently displays a difference scores significantly above zero, 46% to 82% into the movements. Thus, familiar size interference in the known size task cannot be compared among all familiar size categories. 

In the familiar size task (blue) where image size was the interfering factor, difference scores were significantly greater than zero for all categories, 15% to 96% into the movement for category 1, 16% to 88% for category 2, 23% to 81% for category 3, and 22% to 93% for category 4. Thus, there was no overall effect of category in terms of where in the movement interference from image size came online, indicating that the magnitude of the familiar size difference did not significantly impact how congruence effects unfolded. 

Overall then, we replicated the time course effect seen in Experiment 1 (Figure 3C), with image size influencing trajectory earlier in the course of the movement than familiar size. However, when we separated out the categories defined by familiar size difference, we observed no differences in the timing of a congruency effect in the familiar size task and no reliable congruency effect in the image size task. Thus, the magnitude of relative familiar size difference (i.e., category) modulated movements in both the image size and familiar size tasks in terms of trajectory (i.e., maximum curvature) but not in terms of timing. This suggests that differences in the familiar sizes of objects lead to corresponding differences in the degree of interference between image size and familiar size, but not when the interference occurs. This result might also suggest that the conflict resolution process in our paradigm is sensitive during movements and may even be dynamically tailored to this period. 

Summary

The goal of Experiment 2 was to examine whether differing degrees of real-world size difference impact goal-directed pointing to differing degrees. To do so, we expanded our set of real-world objects to include a wider range of familiar sizes and systematically manipulated the magnitude of the difference in familiar sizes between paired targets. Here, we again demonstrated bidirectional interference between the perception of image size and familiar size when the two were presented in an incongruent manner, although we also found evidence of image size exerting a greater influence on familiar size than vice versa. Furthermore, we observed several trends associated with greater magnitudes of familiar size difference between paired objects. Namely, in the familiar size judgment task, the larger the familiar size difference, the lower the maximum curvature of goal-directed pointing movements overall. This is consistent with greater differences in familiar size making familiar size judgments easier. 

Additionally, the incongruence effect in the familiar size task diminished with increasing familiar size difference magnitude. This indicates that image size did interfere with familiar size judgment across categories but did so less when the relative familiar size difference was greater. Similarly, in the image size judgment task, the interference from familiar size increased with increasing familiar size difference across the four categories. Overall our result is consistent with image size perception being the more automatic of the two and exerting greater influence on the decision-making process and on hand movements. However, the greater the difference between the familiar real-world sizes of the objects, the greater the effect that familiar size has and the more it mitigates or interferes with the influence of image size. 

General discussion

Our perception of real-world objects involves conceptualization of how large or small they are in the real world—we expect a rubber duck to be physically smaller than a boat (although artists have taught us that this does not have to be the case) (Konkle & Oliva, 2011; Konkle & Oliva, 2012a; Konkle & Oliva, 2012b). In the current study, we presented participants with pairs of real-world objects and manipulated their relative image size and familiar size. By using a visually guided pointing paradigm and decision-making task based on either image size or familiar size, we were able to assess the relationship between these two aspects of object perception and their impacts on visually guided action. Overall, we observed that the conflict between image size and familiar size is resolved over the course of the movement and results in curved trajectories, suggesting early processing of each feature and a later conflict resolution between the two. 

Mechanisms of image size and familiar size perception

Object perception relies on a hierarchy of perceptual processes representing increasingly complex object features (Riesenhuber & Poggio, 1999). Image size is a low-level feature and is represented essentially by the size of the stimulus on the retina—far earlier in the visual processing pathway than anything related to object identity. However, recent studies have demonstrated that familiar size can be represented and processed earlier than classical models of object processing assume (Collins & Quillian,1969; Jolicoeur et al., 1984; Rosch et al., 1976). 

For example, as early in the image processing pathway as V1, representations have been shown to reflect viewers’ subjective perception of object sizes as opposed to their veridical image size. Sperandio, Chouinard, and Goodale (2012) found that, despite constant retinal image size, V1 activity reflected perceived object size in a size constancy task that manipulated target viewing distance, suggesting that image size perception is influenced by other aspects of perception even at this early stage in processing. 

Furthermore, the familiar sizes of real-world objects have been shown to have neural representations independent of image size. Konkle and Oliva (2012b) showed that familiar object size is represented in the occipitotemporal cortex (OT) much like object categories, with larger objects represented in medial OT and smaller objects in lateral OT. Thus, our observation that both image size and familiar size are processed automatically is consistent with known mechanisms underlying object perception. In addition, recent studies have shown that processing mid-level perceptual features can be sufficient to distinguish objects of different familiar sizes without real-world context (Long, Konkle, Cohen, & Alvarez, 2016; Long, Yu, & Konkle, 2018). 

Here, we report evidence of bidirectional interference that suggests interactions between image size and familiar size mechanisms. Image size was consistently the more dominant of the two—it was processed earlier and more robustly, and participants anecdotally reported finding it more salient than familiar size. That said, we demonstrated in Experiment 2 that the robustness of the influence of familiar size on movements (and we can infer, therefore, its representation strength) increased with increasing familiar size magnitude, and that this robustness mitigated the effects of image size when the two were in conflict. Essentially the more robust process—image size judgment—became less robust and more susceptible to influence from the task-irrelevant feature familiar size when the familiar size difference was greater in magnitude. 

Similarly, we observed that, in the familiar size task, task-irrelevant image size processing interfered less the greater the magnitude of the familiar size difference. Thus, in both tasks, greater familiar size differences more strongly counteracted the influence of image size. This is possibly due to increased salience of familiar size as a feature when the familiar sizes of the two paired real-world objects were more discrepant. Furthermore, the parametric nature of our manipulation in Experiment 2 and the resulting graded effects of familiar size suggest that familiar size may be represented in a continuous manner as opposed to the binary “large object” and “small object” areas found by Konkle and Oliva (2012a). 

Overall, we present converging evidence that image size is processed earlier and more robustly than familiar size but that increasing familiar size differences somewhat mitigate this effect, indicating dynamic interactions between image size and familiar size perception and action. 

Processing conflict in the familiar size Stroop task

The familiar size Stroop task used here was inspired by the classic Stroop task in which identifying the color a word is printed in is impaired when the word is the name of an incongruent color (Konkle & Oliva, 2012a; Stroop, 1935). To accurately perform the Stroop task, participants must suppress the automatic response produced by a direct processing pathway in order to respond to the relevant feature for their current goal, processed by an indirect processing pathway (Botvinick, Braver, Barch, Carter, Cohen, & Botvinick, 2001; Ridderinkhof, van der Molen, & Bashore, 1995; Shenhav, Botvinick, & Cohen, 2013; Van der Stigchel, van Koningsbruggen, Nijboer, List, & Rafal, 2012). 

The prioritization of task-relevant over task-irrelevant features is a key aspect of cognitive control, and the Stroop task has been investigated extensively within the cognitive control literature (Memelink & Hommel, 2013). Resolving conflict in the Stroop task is thought to be comprised of three processes: monitoring conflict between the direct pathway and indirect pathway, adjusting the response threshold by inhibiting motor output, and recruiting top–down processes in order to resolve conflict between the two pathways (Botvinick et al., 2001; Erb et al., 2016; Shenhav et al., 2013). Erb et al. (2016) investigated these individual processes in a classic Stroop task using a visually guided pointing paradigm and showed that they manifested in different aspects of visually guided action. Specifically, movement initiation latency was shown to reflect the response threshold adjustment process and maximum curvature reflected the final conflict resolution between the direct and indirect pathways. 

By using a similar visually guided pointing paradigm and the familiar size Stroop task we are able to speculate about the processing of image size and familiar size and the conflict between the two. When the image size and familiar size of an object were in conflict, participants were slower to initiate movements, and their movements were more curved toward the incorrect response. We propose that the former effect results from motor output being suppressed to allow for more processing time, an early conflict-resolution process. The latter effect suggests that representations of both alternatives persist even after movements are initiated (Erb et al., 2016). Thus, conflict-resolution processes begin early, but conflict is not resolved until later. This also demonstrates that both image size and familiar size, as well as their conflict, influence both movement preparation and movement execution, with a larger effect from image size. 

Again, though, despite the fact that image size influenced familiar size more than familiar size did image size, we did observe bidirectional interference. Although the classical Stroop effect is largely reported in terms of the unidirectional influence of reading a word on reporting a color, a reverse Stroop effect has also been observed in which participants’ ability to read a color name is impaired by incongruent text color (MacLeod, 1991). However, the effect of the written word on printed color identification is more pronounced than the impact of ink color on reading the written word. Thus, interference in the classical Stoop effect is bidirectional but asymmetrical, like our reported effects in the familiar size Stroop. 

Based on these results, processing image size in the familiar size Stroop task parallels reading the word in the classical Stroop task, and familiar size processing parallels reporting the text color. From this we speculate that the perception of the image size of an object is a direct pathway process, and the perception of familiar size is an indirect pathway process, again consistent with participants’ subjective reports of their relative ease. These observations have important implications for target selection, action control, and perception/action integration more broadly. There are a number of reasons why image size processing could be a direct pathway process in this action-based Stroop paradigm. First, image size is arguably more relevant for the guidance of action than familiar size is—we can process the size of an object in front of us and interact with it even if it is not an object we are familiar with, and when the image size and familiar size of a familiar object are incongruent their current image size is far more important for action than their typical familiar size. 

In the classic Stroop task, the direct pathway process is reading a word and the indirect process is naming the color of the ink—again, somewhat counterintuitive given that reading is a higher order process than color perception. However, reading proceeds more automatically due to the relative frequency of performing this learned behavior. Similarly, attending to image size in isolation may be performed more frequently in daily life than processing familiar size in isolation. These two explanations are not mutually exclusive. It is likely that the automaticity of image size processing and its influence on action are the result of its frequency and utility in daily life. 

That said, our results show that familiar size is processed automatically, as well. This learned behavior is more relevant to visual perception than it is to visually guided action in a classical perception/action dissociation framework. However, as we have shown, perception and action are not truly distinct, and perception does influence action. Thus, if image size influences action more readily and familiar size influences perception more readily, this does not mean that familiar size does not influence action, just that the influence emerges later as in other perception/action integration processes. 

It is additionally worth noting that, just as the Stroop effect can be replicated with features other than color names and can inform our understanding of direct and indirect processing pathways, we believe that the action-based Stroop effect seen here is not limited to incongruency between image size and familiar size. Although object size has a significant impact on reach-to-point, as well as other types of hand movements, the conflict revealed in these experiments is as much about cognitive control and conflict resolution as it is about visual perception and action control. Thus, we hypothesize that these results would be replicated with other forms of perceptual incongruency or conflict tasks, and the same asymmetry would be observed as long as one feature represents a direct and the other an indirect pathway. 

Conclusions

By examining the modulation of goal-directed hand movements by the image sizes and familiar sizes of real-world objects, the present study can contribute to a more complete picture of how objects are perceived and identified, how hand movements are guided, and how these processes interact. Comparing the impact of image size and familiar size, image size appears to exert a greater impact and be processed faster than familiar size is. That said, the more complex and difficult perceptual process of judging the familiar sizes of objects, requiring higher level perceptual identification and the recruitment of prior experiences and memory, also occurs automatically and robustly enough to interfere with judgments of image size. Critically, both image size and familiar size are processed even when task irrelevant. 

The strength of this bidirectional influence of image size on familiar size judgment and familiar size on image size judgment is not absolute, however. Experiment 2 demonstrated that when the familiar size difference between paired objects was larger, the effects of familiar size on action became more robust. This had two effects. First, choosing the larger or smaller object based on its familiar size in the real world became easier the larger the difference between the two, despite the fact that a difference was always readily apparent. Second, and more importantly, familiar size interfered with the effects of image size—image size judgments in the image size task and image size interference in the incongruent trials of the familiar size task—more strongly with greater relative familiar size differences, suggesting greater salience and representation robustness. 

Overall, the present study provides evidence that even high-level aspects of visual perception—the identification of real-world objects and the integration of prior knowledge regarding their sizes in the real world—interact with visually guided action automatically and systematically. This points to a far more integrated view of perception and action than classically hypothesized. However, the current study alone is insufficient to explain these interactions on a mechanistic level. Further investigations are needed to identify the mechanisms responsible for the perception of object size (image size, familiar size, and other aspects), decision-making, visually guided action, and their relationships. 

Acknowledgments

Supported by a National Science Foundation Behavioral and Cognitive Sciences award (1849169 to JHS). CMG is supported by a National Science Foundation Graduate Research Fellowship. 

Commercial relationships: none. 

Corresponding author: Joo-Hyun Song. 

Email: [e-mail]. 

Address: Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI, USA.