Visual search for reach targets in actionable space is influenced by movement costs imposed by obstacles

Abstract

Real world search tasks often involve action on a target object once it has been located. However, few studies have examined whether movement-related costs associated with acting on located objects influence visual search. Here, using a task in which participants reached to a target object after locating it, we examined whether people take into account obstacles that increase movement-related costs for some regions of the reachable search space but not others. In each trial, a set of 36 objects (4 targets and 32 distractors) were displayed on a vertical screen and participants moved a cursor to a target after locating it. Participants had to fixate on an object to determine whether it was a target or distractor. A rectangular obstacle, of varying length, location, and orientation, was briefly displayed at the start of the trial. Participants controlled the cursor by moving the handle of a robotic manipulandum in a horizontal plane. The handle applied forces to simulate contact between the cursor and the unseen obstacle. We found that search, measured using eye movements, was biased to regions of the search space that could be reached without moving around the obstacle. This result suggests that when deciding where to search, people can incorporate the physical structure of the environment so as to reduce the movement-related cost of subsequently acting on the located target.

Introduction

Many real-world action tasks involve visual search for target objects that are acted upon once they are located (Land, Mennie, & Rusted, 1999; Land & Hayhoe, 2001). However, few studies have examined whether costs associated with acting on the target object influence the visual search processes that precede the object-directed action. In general, the cost of acting on a target object will depend on where the object is located in the environment. However, the vast majority of experiments examining visual search have used tasks in which the response that participants generate after locating a target (e.g. a button press) is independent of the location of the target in the search space. In other words, most visual search experiments have not been designed to examine whether movement costs associated with acting on a target object can influence visual search. Importantly, movement-related costs might be expected to influence visual search in situations in which there are multiple targets, as when visually searching for one of several knives in a kitchen while cooking. In this scenario, movement-related time and effort costs involved in retrieving the target object (e.g. a knife) can be reduced by first visually searching nearby locations and only then visually searching more distant locations. 

In real-world search tasks, information pertaining to movement costs is often provided by the structure of the environment, including the locations of obstacles that need to be negotiated in order to reach the target object. Thus, in our coffee cup example, the cost associated with a given cup will depend on its location in the cupboard as well as the shapes and locations of other items in the cupboard that the hand has to move around en route to the cup. The aim of the current study was to test the hypothesis that people take into account movement-related costs inherent in the structure of the environment when searching for a target object, even though these costs are only experienced after search has been completed. Note that this hypothesis is broadly aligned with “parallel” models of behavior that posit that cognitive, perceptual, and sensorimotor processing can overlap in time (Cisek & Kalaska, 2010; Gallivan, Chapman, Wolpert, & Randall Flanagan, 2018). 

Visual search studies have considered other ways in which the “structure” of the environment, or visual scene, can influence and inform search. Numerous studies have shown that people use knowledge about objects and environments to determine the likely, and unlikely, locations of a target object in a scene (Antes, 1974; Henderson, Weeks, & Hollingworth, 1999; Henderson & Ferreira, 2004; Eckstein, Drescher, & Shimozaki, 2006; Neider & Zelinsky, 2006; Torralba, Oliva, Castelhano, & Henderson, 2006; Castelhano & Henderson, 2007; Ehinger, Hidalgo-Sotelo, Torralba, & Oliva, 2009; Võ & Henderson, 2009; Võ & Schneider, 2010; Castelhano & Heaven, 2011; Võ & Wolfe, 2013; Pereira & Castelhano, 2014). For instance, according to the Surface Guidance Framework (Pereira & Castelhano, 2019), attention and eye movements are directed to the scene surfaces most associated with a target object. As one example, when searching for a kettle or a dog bowl, gaze may be first directed to countertops or the floor, respectively. It has also been shown that knowledge of a target object's function and where, in a scene, the function is likely to occur, can guide search (Castelhano & Witherspoon, 2016). 

A number of studies have shown that movement-related costs can influence sensorimotor decisions about where and when to move, including very fast decisions related to movement selection made during ongoing action tasks (Cos, Bélanger, & Cisek, 2011; Cos, Duque, & Cisek, 2014; Bakker, Weijer, van Beers, Selen, & Medendorp, 2017; Diamond, Wolpert, & Flanagan, 2017; Brenner & Smeets, 2015; Brenner & Smeets, 2022), Movement-related costs can also influence perceptual decisions; in perceptual discrimination tasks, movement costs associated with responding can bias perceptual judgements (Burk, Ingram, Franklin, Shadlen, & Wolpert, 2014; Marcos, Cos, Girard, & Paul, 2015; Hagura, Haggard, & Diedrichsen, 2017). Movement-related costs have also been shown to influence the extent to which memory is used in a search. For example, Solman and Kingstone (2014) showed that participants made greater use of memory of previous search displays when search required large gaze shifts involving both eye and head movements compared to smaller gaze shifts that could be accomplished with eye movements alone (see also Ballard, Hayhoe, & Pelz, 1995). 

We recently investigated whether movement costs can influence search behavior using reaching tasks in which, after locating a target object presented among distractor objects, participants moved a cursor to a target object (Moskowitz, Berger, Castelhano, Gallivan, & Flanagan, 2022). In one experiment, focusing on movement effort costs, participants controlled the cursor by moving a handle attached to a robotic manipulandum that applied resistive forces that varied across the search space. In another experiment, focusing on movement time costs, participants used a joystick to move the cursor and the speed of the cursor varied across the search space. We found that movement time costs but not movement effort costs could bias where participants searched. However, a limitation of this previous work is that we used artificial manipulations, not commonly experienced during real-world tasks (e.g. variations in cursor speed), that participants had to learn during the experiment. Moreover, in this previous work, the mapping between costs and spatial location in the search space was arbitrary. 

Previous studies have shown that humans rapidly factor in the presence of obstacles when planning reaching and grasping movements (Biegstraaten, Smeets, & Brenner, 2003; Nashed, Crevecoeur, & Scott, 2012; Voudouris, Smeets, & Brenner, 2012; Garzorz, Knorr, Gilster, & Deubel, 2018). Here, we investigated whether people also incorporate the presence of obstacles into their search behavior. Specifically, using a task in which participants were required to reach to a target object after locating it among distractor objects, we asked whether visual search—prior to reaching—takes into account the location of an obstacle that altered the hand path required to reach some of the objects. To address this question, we designed a task in which participants visually searched for one of four target objects among 32 distractors and, once a target was located, reached toward it using a cursor controlled by the handle of a robotic manipulandum. To identify whether an object was a target or a distractor, participants had to fixate at, or close to, its location. At the start of each trial, participants were required to move the cursor to the center of the display. A rectangular obstacle was briefly presented and then hidden from view before the target and distractor objects were presented. Thus, participants had to remember the position of the obstacle. We varied the length, location, and orientation of the obstacle from trial to trial along with the locations of the target objects. Forces applied to the handle of the manipulandum simulated contact between the cursor and the unseen obstacle. To reach to an object located on the far, or “obstructed,” side of the obstacle, the participant was required to move their hand around the obstacle, incurring greater movement time and effort costs relative to reaches to objects located on the near, or “open,” side of the obstacle. We included multiple targets so that movement-related costs could be reduced by initially searching in low-cost regions of the search space (i.e. on the open side). We opted for four targets so that the probability of finding a target in different regions of the search space, including on either side of the obstacle) would be reasonably high (see below). 

We predicted that, in each trial, participants would readily integrate the location of the obstacle into their search strategy, and preferentially search for the target object on the open side of the display. We also predicted that participants would exhibit this search strategy right from the beginning of the experiment, indicating that they could naturally determine movement cost from the structure of the environment without having to learn these costs through experience with the task. 

Methods

Participants

Twelve participants (6 women) between the ages of 18 and 19 years old (M = 18.7) were recruited for this experiment. Participants were required to be right-handed and have normal or corrected-to-normal vision while wearing contacts. All participants were compensated $15 or 1.0 course credits for their participation. Participants provided written informed consent, and after the conclusion of the experiment they were debriefed. The experiment was approved by the Queen's General Research Ethics Board and complied with the Declaration of Helsinki. 

Apparatus and stimuli

Seated participants viewed the visual stimuli—including the target and distractor objects, and a cursor controlled by hand movement—on a vertical monitor positioned directly in front of them (Figure 1). Participants controlled the cursor by grasping the handle of a planar robotic manipulandum (KINARM Endpoint; Kinarm, Kingston, ON, Canada). The position of the cursor on the monitor (filled white circle, radius 3 mm) was linked to the position of the handle grasped by the participant, and moved in a horizontal plane. The direction mapping between handle movement and cursor movement was the same as a standard computer mouse, such that forward and backward movements of the handle moved the cursor up and down, and right and left handle movements moved the cursor right and left. When the cursor was in the center of the screen, the handle was located approximately 20 cm in front of the participant's chest and in the mid-sagittal plane. There was a 1 to 1 correspondence between the distance moved by the handle in the horizontal plane and the distance moved by the cursor on the screen. The position of the handle was recorded at 1000 Hz. Gaze data were collected at a rate of 500 Hz using an infrared eye tracker (Eyelink 1000; SR Research, Ottawa, ON, Canada) mounted just below the display monitor. A chin rest (not shown in Figure 1A) was used to limit head motion during the experiment. 

Figure 1.

 

Experimental setup. (A) Participants moved a cursor to target objects located on a vertical screen by moving the grasped handle of a robotic manipulandum in the horizontal plane. The manipulandum applied forces to the hand if it contacted the obstacle located in the scene. Gaze was recorded with an infrared video-based eye tracker. (B) Objects (squares), obstacle (orange bar), and gaze (black solid trace), and hand cursor (gray dashed trace) paths from an exemplar trial with a long, far obstacle located above the start position (open green circle). (The dashed orange bar shows a short, near obstacle for comparison.) Vision of the obstacle was briefly provided at the start of each trial. After vision of the obstacle was removed, 36 objects were presented on the screen, located in imaginary square cells arranged roughly in a circle. The object position with a cell was randomly jittered from trial to trial. There were four target objects (pink on the left side), and 32 distractor objects (pink on the right side). The dashed horizontal line separates objects considered to be located on the open and obstructed sides of the obstacle. The gray rounded rectangle contains the matched open side objects that mirror the obstructed side objects in this trial (i.e. the objects that would be obstructed if the obstacle was on the opposite side of the search space).

In each search trial, a total of four target and 32 distractor objects were presented on the screen, located within imaginary 3.8 × 3.8 cm square cells. The cells were located in one of seven columns and one of seven rows and roughly arranged in a circle (see Figure 1B). The position of each object within the cell was jittered using a random shift in the x and y location drawn from a continuous distribution between ± 0.7 cm in order to create unique object locations across trials. The start position for the hand (empty green circle, radius 5 mm) was located in the center of the display. Target and distractor objects were 1 cm wide squares (subtending approximately 1.6 degrees of visual angle when in the center of the monitor) split vertically down the middle with one half colored pink and the other half blue. The target objects had the opposite color arrangement to the distractor objects and the color arrangement for the targets (and distractors) was counterbalanced across participants. Participants were informed about the appearance of the targets and distractors at the start of the experiment. Critically, identifying an object as either a target or a distractor required fixating on, or close to, its location (Moskowitz et al., 2022). 

An obstacle (orange rectangle) was positioned in one of eight locations. Specifically, the obstacle could be located in one of four directions—in front of, behind, to the left, or to the right—relative to the start position of the hand in the horizontal plane. Thus, on the screen, the corresponding obstacle location was above, below, to the left, or to the right of the start position of the cursor. In addition, the obstacle was located at one of two distances from the start position (close and far). The obstacle was oriented vertically when positioned to the left or right, and horizontally when positioned above or below the start location. When the obstacle was at the close and far positions, the inside edge of the obstacle (closest to the start position) was 1.5 or 5.3 cm from the center of the start position, respectively. Finally, the obstacle could be one of two lengths, either long (20 cm) or short (5 cm). The width of the obstacle was always 0.8 cm. Thus, there were a total of 16 possible obstacle configurations that could appear on any given trial. The orange rectangle in Figure 1B shows the long obstacle positioned in the “far” position above the start location. For comparison, the dashed orange rectangle shows the short obstacle in the close position, also above the start location. The edges of the obstacle were modeled as a very stiff spring (6000 N/m) with damping (-4 Ns/m) that prevented the handle of the manipulandum from crossing the outer edges of the obstacle, simulating a physical barrier placed in the search space. To select target locations, we randomly sampled without replacement four target locations from the 36 possible target locations. To select the obstacle configurations, for each block of 16 experimental trials, we randomly permuted all 16 possible obstacle configurations and assigned those configurations to that block of trials. 

Procedure

At the beginning of each trial, the participant moved the cursor to the start position and, after a delay of 750 ms, a fixation cross (white, width 1.4 cm) appeared on top of the start position that the participant was required to fixate. After 500 ms, the obstacle was displayed for 1000 ms and, after an additional delay of 1000 ms, the fixation cross and start position were removed from view and the target and distractor objects were presented. At this point, the participant was free to move their gaze while searching for a target object. The obstacle was removed from view so that, visually, there was no difference between regions of the display on the open and occluded sides of the obstacle. Critically, the forces simulating the obstacle remained throughout the trial. Thus, although the obstacle was removed from view, it was nevertheless still present. 

Once the participant located one of the four target objects, they made a reaching movement to move the cursor to that target. The reach was completed when the center of the cursor was within 3 mm of any part of the target object—such that the cursor overlapped with the target—for 100 ms. There was no movement time criterion but participants were instructed to leave the cursor at the start location until they located a target object. When the cursor reached a target object, the phrase “TARGET FOUND” was presented on the monitor and a “correct” tone (5000 Hz, 100 ms) was played. If a target object was not located within 30 seconds after object presentation, the phrase “TIMEOUT” was presented on the monitor and an “incorrect” tone (5 Hz, 100 ms) was played. This occurred in less than 1% of all trials. Feedback remained on the screen for 1500 ms before the next trial began. 

Before beginning the experiment, we ran participants through an eye-tracking calibration. Participants were informed that there would be four target objects on each trial and that their locations were randomized. Participants completed four familiarization trials in which the obstacle was visible throughout the trial and participants were encouraged to contact the obstacle so that they could appreciate that the obstacle acted as a physical barrier to their hand and the cursor. Participants then completed a total of 208 search trials, with a short rest inserted every 30 trials. 

Data analysis

After eliminating blinks, the raw gaze signal was smoothed using a second-order, zero-phase lag Butterworth filter with a cutoff frequency of 50 Hz. We then extracted fixation locations for each trial from the time at which the target and distractor objects were presented until the time that a target was reached by the cursor. In each trial, we identified the fixation of the start position and the fixation of the target during the reach, and used these fixations for drift correction in both the x and y gaze positions. Figure 1B shows the drift-corrected gaze signal for an exemplar trial with fixation locations highlighted in blue and numbered. 

For each trial, we attempted to assign each fixation location (other than the initial fixation at the fixation cross) to an object location. Specifically, we assigned each fixation to the closest object, as long as it was no more than 1.5 cm in distance from the center of that object. Fixations that could not be assigned to an object were removed from the analysis. This was infrequent, as less than 1% percent of all fixations could not be assigned to an object. Two or more successive fixations, separated by small saccades, could occasionally occur at a given object. In these cases, we only extracted the first fixation. 

In our analysis we focused on the effects of obstacle position (far versus close) and length (short versus long) on search and reach behavior, collapsing across obstacle angle (above, left, right, or below the start). This was done by rotating the fixation and object locations to a common coordinate frame with the obstacle always on the left side of the space. We then defined any objects/fixations to the right of the obstacle as being on the “open” side of the search space, and objects/fixations to the left of the obstacle as being on the “obstructed” side of the search space. Although for short obstacles, unlike long ones, participants could reach some of the objects in the first column to the left of the obstacle without deviating their reach from a straight line, our initial data analysis showed that participants treated these objects as being on the obstructed side of the obstacle (see below). 

To analyze movement duration and distance, we defined movement onset as the time that hand speed exceeded 5 cm/s or, if the hand did not reach that speed (which could sometimes happen when a target was located very close to the start position), when the cursor exceeded a distance of 1 cm from the start location. Movement offset was defined as the time the target was reached. 

We expected that when reaching to targets on the open side, participants would move their hand in roughly a straight line to the target. Conversely, when reaching to targets on the obstructed side, we expected participants to move their hand around the obstacle, provided that they remembered its location. To examine that initial direction of the hand movement, we computed, for all trials, the “absolute reach deviation angle,” defined as the absolute angle between the initial reach direction and the straight line from the position of the cursor at the start of the trial (cursor start position) and the center of the target (see inset in Figure 2A). The initial reach direction was taken as the vector from the cursor start position to the position of the cursor when it had moved 1 cm from the cursor start position. 

Figure 2.

 

Reach and eye movement characteristics. (A) Cumulative distributions of absolute reach deviation angle (see inset and text for details) for reaches to the open and obstructed sides, with separate curves shown for each obstacle condition (see legend in E). (B) Hand paths from all trials across all participants to obstructed side targets with the long, far obstacle. To allow comparison across the different possible object locations, the paths are rotated so that the obstacle is on the left. Red and green traces show trials in which the obstacle was or was not contacted, respectively. (C, D) Box plots for movement time (C) and distance (D) for each side and obstacle condition. The lower and upper edges of each box represent the first and third quartiles, the horizontal line represents the median, and the whiskers extend to the smallest and largest data points within 1.5 times the interquartile range below and above the first and third quartiles. Data points outside the boundary of the whiskers are shown as circles. (E, F) Distributions of the distance between successive objects fixated during search (E) and the number of objects fixated in a trial (F), with separate distributions shown for each obstacle condition. (E) The dashed vertical line presents the average distance (3.8 cm) between adjacent objects in the same row or column of the search display.

Repeated measures ANOVAs were used to test our main hypotheses. We used an alpha value of 0.05. 

Results

Reach movements to the open and obstructed sides

Our hypothesis that visual search would be biased toward objects on the open side of the search space assumes that participants would remember the location of the obstacle and move around it when reaching to targets found on the obstructed side of the search space. To assess whether this is the case, we compared the absolute reach deviation angle in reaches to objects on the obstructed side to the absolute reach deviation angle in reaches to “matched” objects on the open side (see the gray region in Figure 1B); that is, the objects on the open side that would—in a given trial—have been on the obstructed side had the obstacle been located on the opposite side of the search space (i.e. in the mirrored location). Figure 2A shows, for each obstacle condition and side of the search space (obstructed side versus match open side), cumulative distributions of the absolute reach deviation angle for all trials from all participants (note that the legend in Figure 2E applies to all panels except B). As expected, we found that reaches to open side targets were initially aimed in the approximate direction of the target; indeed, in the large majority of reaches, the absolute reach deviation angle was less than 30 degrees and the median angle was approximately 10 degrees. Conversely, we observed much larger absolute reach deviation angles for reaches to obstructed side targets, and the median angle was approximately 60 degrees. However, the absolute reach deviation angle was less than 30 degrees in a substantial number of these reaches, and we found that participants contacted the obstacle in 58.2 ± 5.2 % (M ± SE) of reaches to obstructed side targets. 

To provide a sense of the participants’ behavior, in Figure 2B, we have plotted hand cursor paths—from all trials performed by all participants—with the long, far obstacle (shown as an example) in which the reach was directed to a target on the obstructed side. The red and green traces show trials in which the obstacle was contacted or successfully avoided, respectively. As illustrated in the figure, in most trials, participants attempted to reach around the obstacle, although they failed to avoid contact on many of these attempts. However, on a substantial proportion of trials, participants made straight line movements directly into the obstacle, suggesting that they either failed to encode the location of the obstacle when it was briefly presented at the start of the trial, or forgot its location while searching for a target object. Note that the presence of trials in which the obstacle location was not encoded or forgotten works against our hypothesis, weakening the chances of observing a search bias to the open side. 

Figures 2C and D show boxplots for movement time (MT) and movement distance (MD) for each obstacle condition and each side of space. Both MT and MD were larger when reaching to the obstructed side compared to the open side. For reaches to targets on the obstructed side, both MT and MD increased with obstacle length and distance from the start position. To quantify these effects, we carried out separate 2 (obstacle length: long, short) × 2 (obstacle position: far, close) × 2 (side: matched open side, obstructed side) repeated measures ANOVA for MT and MD, using the average MT and MD for each combination of length, position, and side. For MT, we found significant effects of side (F(1, 11) = 292.99, P < 0.001, η² = 0.964), obstacle length (F(1, 11) = 66.40, P < 0.001, η² = 0.858), and obstacle position (F(1, 11) = 9.52, P = 0.01, η² = 0.464). Similar results were obtained for MD. Specifically, we found significant effects of side (F(1, 11) = 478.18, P < 0.001, η² = 0.978), obstacle length (F(1, 11) = 111.01, P < 0.001, η² = 0.910), and obstacle position (F(1, 11) = 25.03, P < 0.001, η² = 0.695). When collapsing across obstacle conditions, the average MT when reaching to targets on the occluded side (M = 2.00 seconds and SE = 0.29 seconds) was over twice as long as when reaching to targets on the open side (M = 0.87 seconds and SE = 0.18 seconds). 

General gaze behavior during the search

Based on our previous work (Moskowitz et al., 2022), we expected that, during search, the majority of gaze shifts would be between adjacent or nearby objects. Figure 2E shows distributions, including all data from all participants, of the distance between successive object fixations—defined as the distance between the centers of the objects fixated—for each obstacle condition. Similar distributions were observed across all obstacle conditions, with the largest peak around 3.8 cm, which is the average distance between adjacent objects in a row or column (see vertical dash line in Figure 2E). The smaller peaks at each multiple of this distance represent cases in which gaze shifted to an object located two or three objects away from the currently fixated object. Thus, participants mostly fixated neighboring objects when searching. We also examined the number of objects, including the target object, that participants fixated in each trial (Figure 2F). Across all trials and participants, we found that, on average, there were 6.12 object fixations, including occasional re-fixations. On average, there were 0.16 object re-fixations per trial. 

Participants prefer to fixate objects on the open side

To test our main hypothesis, we examined the location of participant fixations across trials to determine whether there was a bias to visually searching the open side of the search space. Figures 3A to D show the locations of all fixations, across all participants and trials, in each obstacle condition. Note that fixations associated with the four different obstacle angles are shown in a common space with the obstacle on the left side of the start location. To create this visualization, we took all fixations that were assigned to an object location (i.e. those within 1.5 cm of an object) and then, for each fixation, adjusted its location to compensate for the random shift applied to the object it was assigned to on that trial. In this way, we were able to plot fixations across trials aligned to a common, object-centered grid. 

Figure 3.

 

Distribution of fixations across the search space. Fixation locations from all trials and participants for (A) far and long, (B) close and long, (C) far and short, and (D) and short and close obstacle conditions. Fixations were assigned to the closest search object, with the fixation location being shifted by the amount of jitter applied to that assigned object. Fixation and obstacle locations were rotated to a common reference frame with the obstacle on the left side of the search grid. The histograms at the left and right show, for each row and column in the search display, respectively, the average number of fixations per object in that row or column.

What can be quickly appreciated (see Figure 3) is the asymmetry in fixation density between the obstructed and open sides across all four obstacle conditions, with a higher density of fixations on the open side. The histograms at the bottom of the figure show, for each obstacle condition, the average number of fixations per object in each column (i.e. the total number of fixations in that column divided by the number of objects in that column). As might be expected, the asymmetry in the distribution is greater for the close object conditions than the long object condition. However, similar distributions were observed for short and long obstacles. The histograms at the left of the figure show, for each obstacle condition, the average number of fixations per object in each row. These distributions are similar across all obstacle conditions. 

To further examine the density of fixations per column, we plotted, in Figure 4A, the average number of fixations per object per column for all four obstacle conditions. We defined the center column containing the start location as zero, with the columns to the right of it being assigned positive integer values, and the columns to the left being assigned negative integer values. We refer to the magnitude of these integers as the column's eccentricity (i.e. distance from the center). To assess the influence of the obstacle on search behavior we conducted a 2 (obstacle position: far, close) × 2 (side: left, right) × 3 (eccentricity: 1, 2, and 3) repeated measures ANOVA on the average number of fixations per object in each column. We did not include the center column as a level of eccentricity because we are mainly interested in testing whether participants showed a bias toward a side of the search space. We chose to leave out obstacle length as a factor in this analysis because it did not appreciably impact fixation density (as shown in Figure 3). 

Figure 4.

 

Distribution of fixations across space and time. (A) Average number of fixations per object, in a given trial, for each column of the adjusted search space (with the obstacle on the left). (B) Fixation x-position, in the adjusted search space, of successive fixations (up to 10) within a trial. (A, B) Separate curves are shown for each obstacle condition and the green line represents the unbiased model. The dashed black lines represent the locations of the near and far obstacles. Curves are based on participant means and the error bars represent ± 1 standard error. (C, D) Black curves show distribution of distances between successive fixated objects (C) and number of objects fixated per trial (D) combining all trials from all participants. Green traces show corresponding model data. (C) The left vertical dashed line represents the average distance between adjacent objects in the same row or column and the right vertical dashed line represents the average distance between adjacent objects that are one row and one column apart. (D) The black and green vertical dash-dotted lines show the medians of the actual and model distributions.

Critically, we found a main effect of side (F(1, 11) = 13.61, P = 0.004, η² = 0.553), with participants fixating more than twice as often on the side without an obstacle (M = 0.18 and SE = 0.02) when compared to the side of the display where the obstacle was located (M = 0.08 and SE = 0.01). In addition, we found an effect of eccentricity (F(1, 11) = 27.41, P < 0.001, η² = 0.714), with participants decreasing the number of fixations at larger eccentricities (see Figure 3A). There was also a significant three-way interaction between position, side, and eccentricity (F(1, 11) = 8.73, P = 0.002, η² = 0.442), which was likely driven by the difference in fixation density in the -1 eccentricity column between close (M = 0.10 and SE = 0.01) and far obstacle trials (M = 0.15 and SE = 0.02). Finally, we observed a significant effect of position (F(1, 11) = 29.52, P < 0.001, η² = 0.729), with participants making slightly more fixations per object in far obstacle trials (M = 0.14 and SE = 0.01) than in close obstacle trials (M = 0.13 and SE = 0.01). No other two-way interactions were significant (P > 0.05 in all three cases). We also observed a clear asymmetry between the −1 and +1 eccentricity columns in terms of fixation density, even in far obstacle trials, in which both of these columns are located on the open side of the display. This result suggests that participants tended to stay away from the obstacle side during search, even when targets located between the start position and the obstacle were unobstructed and could therefore be easily reached. 

Figure 4B shows the average x-position (in the common space), based on participant means, of the first 10 fixations. As can be visually appreciated, participants tended to first fixate a target close to the central start position on the open side and then shift their gaze further away from the obstacle. After five or six fixations, if a target object had not yet been found, they then tended to shift their gaze back toward the midline. This pattern of results is consistent with our finding that participants tended to make small gaze shifts. The black traces in Figures 4C and 4D show the distribution of the distance between successively fixated objects and the distribution of the number of object fixations. These distributions are similar to those shown in Figures 2C and 2D except that they include all trials from all participants and combine obstacle conditions. The left and right vertical dashed lines in Figure 2C shows the average distances between adjacent objects in the same row or column and adjacent objects on the diagonal (i.e. one row and one column away), respectively. The dashed black line in Figure 4D represents the average number of objects fixated across all trials (5.45 objects). (The green dashed line is described below.) 

Given that gaze was initially at the central start position in each trial, and the fact that participants typically shifted their gaze to adjacent objects while searching, we would expect that gaze would be slightly biased to the open side even if participants ignored the obstacle. 

To estimate this bias, we implemented a simple model of search that captured participants’ tendency to shift gaze to nearby objects but that did not take the position of the obstacle into account. During search, the model assigned, to each available (i.e. unvisited) object, a probability that this object would be selected (i.e. fixated) next. The model started with gaze at the start position and each selection made by the model considered the distance of each available object (j) from the currently fixated object (i), defined as d_ij. The cost, C_j, of choosing object j as the next object to fixate was simply the distance to that object, d_ij:  

\begin{equation*}{C_j} = {d_{ij}}\end{equation*}

 

The model assumes there is noise in the decision making process (or calculation of this cost) such that objects with higher costs are sometimes selected. To represent this noise in object selection, we used a softmax selection rule (Wichmann & Hill, 2001) to translate costs into probabilities. Thus, the probability of selecting object j is defined as:  

\begin{equation*}{{\rm{P}}_{\rm{j}}} = {\rm{ }}{{\rm{e}}^{ - {\rm{BCj}}}}{\rm{/ Sum(}}{{\rm{e}}^{ - {\rm{BCj}}}}{\rm{)}}.\end{equation*}

 

The parameter β determines the noise in decision making. In general, the probabilities given to the available objects by the softmax function are ordered according to the cost (with the highest probability for the lowest cost). Thus, although the low cost objects are more likely to be selected, higher cost objects can occasionally be selected. 

For a range of values of the parameter β, we ran 10,000 simulated trials where the locations of the distractor and target objects were taken from a randomly sampled trial from trials (N = 208 trials per participant × 12 participants = 2496 trials) used in the actual experiment. Note that each simulated trial terminated when a target object was fixated. Using this approach, we determined the value of β that minimized the difference between the actual distribution of distances between successively fixated objects (combining data from all participants) and the distribution generated by the model. To compute this difference in the distributions, we determined the proportions of simulated and actual distances in 0.5 cm bins and summed the absolute differences in these proportions across bins. The best fit value of β was 0.558. 

Although this preliminary model was able to closely approximate the actual distribution of distances between successively fixated objects, it overestimated the average number of objects fixated per trial. This suggests that participants were sometimes able to detect the target object while fixating an immediately adjacent object. We therefore fit a detection parameter that specified the probability of a target object being detected when fixating an immediately adjacent distractor object in the same row or column. (i.e. objects that were 3.8 cm, on average, from the target object). For different values of this detected parameter, P_d, we ran 10,000 simulated trials (as described above) using the best fit value of β. Using this approach, we determined the value of P_d that minimized the absolute difference between the simulated and actual average number of objects fixated per trial. The best fit value of P_d was 0.090. 

The green traces in Figures 4A to 4D represent the output from the model. As shown in Figure 4C, the model, as expected, captures the strong tendency to make gaze shifts to adjacent objects. As illustrated in Figure 4D, the model does a very good job at capturing the distribution of the number of objects fixated per trial. The average number of fixations in the model was 6.13, which is very similar to the actual average number of fixations (6.12 as reported above). As shown in Figure 4A, the model captured the overall eccentricity effect observed in the data but, as expected, was not biased toward objects on the open side. 

To further assess the bias in search behavior linked to the location of the obstacle, for each participant and obstacle condition, we separately computed the mean number of fixations per object (averaging across trials) for objects located on the open and obstructed sides. Note that this measure allows us to compare the frequency of fixations on the open and obstructed sides while taking into account the different number of objects on the two sides. Figure 5A shows the average number of fixations per object on the open and obstructed sides for each obstacle condition. The gray lines represent individual participants and the horizontal green lines show the results of the model. The model predicts more fixations per object on the open side because the initial fixation point is on the open side and the gaze shifts are almost always to adjacent objects. However, our results—both at the group level and at the level of individual participants—suggest that this difference is much greater than is predicted by the model. 

Figure 5.

 

Search bias as the group and individual participant levels. Average and individual search. (A) Average number of fixations per object on the open and obstructed sides of the search environment for each obstacle condition. Thin gray lines represent individual participants. (B) Ratio and difference between the average numbers of fixations per object on the open and obstructed sides, shown for each obstacle condition. (C) Proportion of object fixations on the open side as a function of trial block. Thin lines represent individual participants. The thin red line shows the only participant who did not exhibit a significant search bias. (A-C) Green lines represent predictions of the unbiased search model. (B, C) *, **, and *** represent probabilities (P < 0.05, P < 0.01, and P < 0.001, respectively) for one sample t-tests with the model prediction as the reference value.

Figure 5B shows both the ratio (open bars) and difference (hatched bars) between the average number fixations per object on the open side and the average number fixations per object on the obstructed side. The green lines show the corresponding model predictions. We used single sample t-tests to compare the ratio and difference for each obstacle type to the predicted model value and, in all cases, we observed a significant difference (P < 0.05 in all cases). As shown by the ratios, across the different obstacle conditions a given object was approximately three to four times more likely to be fixated if it was located on the open side compared to the obstructed side. These results provide further evidence that visual search was biased toward objects on the open side of the search space and thus took the location of the obstacle—seen prior to but not during search—into account. 

To assess how the bias in visual search behavior evolved over trials, we determined, for each participant, the proportion of all object fixations that were directed to objects on the open side for each successive trial block. Note that each block consisted of 16 trials with each of the 16 possible obstacle configurations: four angles (up, down, left, and right), two distances (close and far), and two lengths (short and long). The thick black line in Figure 5C shows the proportion of object fixations on the open side as a function of block number. The thin gray lines represent individual participants and the horizontal green line shows the prediction from the model. A repeated measures ANOVA revealed that the proportion of object fixations on the open side varied across blocks (F(15, 165) = 2.80, P < 0.001). However, a linear trend was not observed (F(1, 11) = 2.80, P = 0.122), and a separate linear regression failed to show that the proportion of object fixations on the open side depended on block number (F(1, 14) = 0.68, P = 0.421, R² = 0.047). These results indicate that the bias in visual search toward the open side was consistently observed throughout the experimental session, suggesting that participants could immediately appreciate the movement costs associated with the obstacle. Thus, these findings support our hypothesis that participants would be able to naturally determine movement cost from the structure of the environment (i.e. obstacle location in the search space) without having to learn these costs through experience with the task. 

Participants prefer to reach to targets on the open side

Given that fixations were biased toward objects on the open side, we would also expect located targets, and hence reaches, to be biased toward the open side and away from the obstructed side. Indeed, a linear regression analysis indicated that the proportion of reaches to open side targets was strongly predicted by the proportion of open side fixations (F(1, 11) = 123.2, P < 0.001, R² = 0.93). 

We found that participants reached toward the obstructed side in 15.4% of far obstacle trials (SE = 1.6%, 95% confidence interval [CI] = 12.3% to 18.5%) and 23.1% of close obstacle trials (SE = 3.0%, 95% CI = 17.3% to 28.9%). As might be expected, the proportion of reaches to the obstructed side was smaller for far obstacle trials, in which the proportion of objects (and therefore the average proportion of targets) on the obstructed side was 22.2%, than for near obstacle trials, in which the proportion of objects was 41.6%. Note that in only 0.1% and 2.3% of all far and near obstacle trials, respectively, were all four targets on the obstructed side, requiring the participant to search on the obstructed side. Thus, although participants were biased to targets on the open side (as noted above), they nevertheless located, and then moved to, targets on the obstructed side far more often than was strictly necessary. In 90% of close obstacle trials and in 65% of far obstacle trials, at least one target was located on the obstructed side. (Thus, no targets were on the obstructed side in 10% of close obstacle trials and 35% of far obstacle trials.) Therefore, if a participant opted to search on the obstructed side in a given trial, there was still a reasonably high probability that they would find a target on that side. In other words, the bias we observed was not dictated by the task. 

Given the bias to search on the open side, the question arises as to whether participants tended to first search on the open side in trials in which they both remembered the location of the obstacle and reached to a target on the obstructed side. To examine this question, we categorized trials in which the reach target was on the obstructed side as either “remembered” trials—in which the participant either reached around the obstacle, without hitting it, or attempted to reach around the obstacle. Guided by the data shown Figure 2A, we operationally defined remembered and forgotten trials as those in which the participant reached toward the obstructed side with an absolute reach deviation angle >40 degrees and ≤40 degrees, respectively. We defined “control” trials as those in which participants reached to matched targets on the open side (as defined above) with an absolute reach deviation angle ≤40 degrees (which included the large majority of trials with matched reach targets). We found that the average number of fixations in remembered trials (M = 10.26 and SE = 1.23) was nearly twice as great as in control trials (M = 5.48 and SE = 0.22), and was also greater than in forgotten trials (M = 7.14 and SE = 0.58). Note that all pairwise comparisons had large effects sizes (Cohen's d > 1.016 in all 3 cases). The slightly larger number of fixations in forgotten trials compared to control trials may be due to misclassification of a few forgotten trials as remembered trials. Motivated by these results, we determined, for remembered trials, the average number of fixations on the open side that preceded the first fixation on the occluded size. We found that, on average, participants made 6.84 (SE = 1.12) initial open side fixations, which is greater than the average number of fixations per trial across all trials. This result is consistent with our finding that search is biased toward targets on the open side. 

Discussion

The goal of the current study was to examine whether visual search for a target that will subsequently become an action goal, is influenced by movement costs associated with the structure of the environment (e.g. the location of a couch that one would have to walk around into order to pick up a located search object). Specifically, we hypothesized that when searching for a target object—located among distractor objects—in an environment featuring an obstacle, participants’ search would be biased toward locations that can be reached without having to move around the obstacle. We also hypothesized that this bias would be observed throughout the experiment and would not require experience performing the task. In our experiment, the open and obstructed sides differed in area, the number of contained objects, and the average location of these objects. Therefore, to fairly assess potential bias in search, we compared actual search behavior to the behavior of an unbiased search model. Importantly, this model was able to emulate participants’ general search behavior in terms of the distribution of distances between successively fixated objects and the distribution of the number of fixations within a trial. We found clear support for both of our hypotheses. Overall, we found that, in comparison to the unbiased model, participants were more likely to fixate objects on the open side and less likely to fixate objects on the obstructed side. We also found that this bias was present throughout the experiment. 

In a previous study, we investigated the influence of time and effort costs, associated with moving a cursor to a located target, on visual search (Moskowitz et al., 2022). Time costs were manipulated by varying the gain between joystick motion and cursor speed across the search space, and effort costs were manipulated by varying the resistive force applied to the reaching hand across the search space. We found that time costs produced a small but statistically reliable bias on visual search whereas effort costs had no effect. Notably, both of these manipulations involved an arbitrary mapping between movement-related costs and spatial location that could only be learned through experience. In contrast, in the current study, we demonstrate that the presence of an obstacle in the reachable search space results in a bias on visual search that is both strong and immediate in the sense that the bias was observed in the first block of trials. This suggests that participants could readily appreciate the movement related costs associated with the obstacle and integrate this information when making decisions about where to search. It may be worth noting that participants’ decision making was not categorical such that they exclusively searched the open side, even though this would have been a successful strategy (because at least one target was located on the open side in over 98% of the trials). Thus, the search bias appeared to arise from a heuristic strategy rather than a rule-based strategy. Note that in the experiment from our previous study (Moskowitz et al., 2022) examining movement time costs on visual search, the maximum cursor speed when reaching to targets located on the “fast side” of the search space was 2.67 times greater the maximum speed when reaching to targets located on the “slow side” of the search space. In the current study, reaches to open side targets were, on average, 2.3 faster than reaches to occluded side targets. Thus, it does not appear that the difference between the two studies in terms of the influence of movement-related costs on visual search can be attributed to the time cost advantage of reaching to the “faster” side. 

Because targets were randomly located in our task, the optimal search strategy—considering movement-related costs—is to exhaustively search the open side before switching to the occluded side if necessary. However, our participants searched on the obstructed side in approximately 20% of all trials and therefore did not consistently use this strategy. We found that in trials in which participants reached around the remembered obstacle to a target on the occluded side, the number of objects they fixated on the open side, before fixating an object on the occluded side, was slightly greater than average number of objects fixated per trial across all trials. It is tempting to relate this search behavior to patch foraging behavior where, according to the marginal value theorem, foragers should shift to a new patch (e.g. the occluded side) when the reward rate of the current patch (e.g. the open side) drops below the average reward rate across patches learned through experience (Charnov, 1976). However, because our task did not require participants to shift between patches (i.e. sides), and participants often searched only on the open side, we cannot estimate the average reward rate at which participants switched search from the open side to the occluded side. Interestingly, previous work has provided partial support for the hypothesis that human search behavior in visual patch foraging tasks can be accounted for by optimal foraging theory (e.g. Cain, Vul, Clark, & Mitroff, 2012; Wolfe, 2013; for review see Kristjánsson, Björnsson, & Kristjánsson, 2020). 

Participants’ preference to search the open side may have been due to several different movement related costs. Not surprisingly, we found that reaching around the obstacle resulted in increased movement time as well as increased movement distance. Thus, both movement related time and energy costs may have influenced decisions about where to search. However, search decision making may have also been influenced from movement planning and control costs. Wong, Goldsmith, and Krakauer (2016) found that preparing curved reaching movements that navigate paths around obstacles incurs a large reaction time cost in comparison to preparing unobstructed point-to-point reaching movements. These authors suggested that when the path of a movement is task relevant, motor planning involves an additional stage involving the representation of the desired movement path. Reaching around an obstacle may also involve an increased cost of control. It has been suggested that exerting control to improve motor precision comes at a cost involved in attenuating intrinsic neural noise (Manohar, et al., 2015). When reaching directly to a target, precision is only required at the end of the movement whereas, when reaching around an obstacle, some degree of precision is also required at the point at which the hand navigates around the obstacle's edge. The search bias toward the open side may arise from an aversion to crashing into the obstacle, which is a risk when reaching to the obstructed side. Our findings can be related to work proposing that people select gaze targets in order to provide task-relevant information that is rewarding to them (Hayhoe & Ballard, 2014; Tong, Zohar, & Hayhoe, 2017; Zhang, et al, 2018). In our task, expected reward is linked to action costs associated with different potential gaze targets, where lower costs can be considered to be more rewarding. 

Work on sensorimotor decision making—including choices about which movement to make and how fast to make it—has shown that both effort costs and rewards are temporally discounted (Shadmehr, Jacques Orban de Xivry, Xu-Wilson, & Shih, 2010; Rigoux & Guigon, 2012; Berret & Jean, 2016). That is, the influence costs and rewards have on decision making decreases with the delay between when decisions are made and when movement related costs and rewards are incurred. However, we observed that movement related costs influence decisions about where to search even though these costs are incurred after search has been completed. Nevertheless, it may be that movement related costs would have less influence on search decisions if the time required to locate a target was, on average, greater. 

Our results build on previous studies showing that movement related costs associated with searching—as opposed to actions performed following search—influence how memory resources are used during search. Thus, when visual search requires large gaze shifts involving both eye and head movement, participants made greater use of memory of previously presented search displays (to speed search) in comparison to when search can be accomplished using smaller gaze shifts involving eye movements alone (Solman & Kingstone, 2014). Moreover, when search involves walking around an environment to locate target objects, people form long-term memories of the location of objects in the environment as well as the structure of the environment so as to speed up search (Kit, Katz, Sullivan, Snyder, Ballard, & Hayhoe, 2014; Li, Aivar, Kit, Tong, & Hayhoe, 2016; Li, Aivar, Tong, & Hayhoe, 2018). In addition, people are better at avoiding previously searched locations when search involves locomotion in comparison to eye movements (Gilchrist, North, & Hood, 2001; Smith, Hood, & Gilchrist, 2008). These results suggest increasing reliance on spatial memory is beneficial for minimizing energetic costs associated with search (Solman & Kingstone, 2014; Li et al., 2018). 

Our results also build on work showing that movement related costs can influence decision making outside the domain of sensorimotor control. Hagura and colleagues (2017) examined a perceptual discrimination task in which participants indicated whether a random-dot stimulus was moving to the left or right by moving either the left or right hand. Resistive forces could be applied to both hands and the authors found that perceptual judgments were biased toward the direction associated with the hand requiring less effort to move. The authors also found that the perceptual bias, acquired from repeatedly performing the task with resistance applied to a selected hand, persisted when participants switched from using their hands to using verbal responses to indicate motion direction. These results, like the results presented in the current paper, challenge serial models of behavior that posits that movement planning and control occurs after, and is independent from, perceptual and cognitive processing (Miller, Eugene, & Pribram, 1960; Sternberg, 1969; McClelland, 1979). Instead, the results are broadly consistent with parallel models of behavior that hypothesize that processes involved in perception, cognition (e.g. search decision making) and action can overlap in time (Cisek & Kalaska, 2010; Gallivan, Barton, Chapman, Wolpert, & Flanagan, 2015; Gallivan, Logan, Wolpert, & Flanagan, 2016; Pezzulo & Cisek, 2016; Gallivan, Stewart, Baugh, Wolpert, & Randall Flanagan, 2017; Gallivan et al., 2018). 

In conclusion, we note that our results resonate with previous work on visual search showing that the structure of the environment, well as the mapping between structure and function, informs visual search behavior (e.g. Castelhano & Witherspoon, 2016; Draschkow & Võ, 2017; Pereira & Castelhano, 2019). Whereas much of this previous work has focused on where target objects are likely located, the current results focused on where participants would prefer to look given costs associated with moving to a target object once it is found. 

Acknowledgments

Supported by funds provided by the Natural Sciences and Engineering Research Council of Canada and the Canadian Institutes of Heath Research to J. Randall Flanagan. 

Commercial relationships: none. 

Corresponding authors: Joshua B. Moskowitz and J. Randall Flanagan. 

Emails: [e-mail], [e-mail]. 

Address: Centre for Neuroscience Studies, Queen's University, Botterell Hall, 18 Stuart Street, Kingston, Ontario K7L 3N6, Canada.