Contextual factors determine the use of allocentric information for reaching in a naturalistic scene

Abstract

Numerous studies have demonstrated that humans incorporate allocentric information when reaching toward visual targets. So far, it is unclear how this information is integrated into the movement plan when multiple allocentric cues are available. In this study we investigated whether and how the extent of spatial changes and the task relevance of allocentric cues influence reach behavior. To this end, we conducted two experiments where we presented participants three-dimensional–rendered images of a naturalistic breakfast scene on a computer screen. The breakfast scene included multiple objects (allocentric cues) with a subset of objects functioning as potential reach targets (i.e., they were task-relevant). Participants freely viewed the scene and after a short delay, the scene reappeared with one object missing (target) and other objects being shifted left- or rightwards. Afterwards, participants were asked to reach toward the target position on a gray screen while fixating the screen center. We found systematic deviations of reach endpoints in the direction of object shifts which varied with the number of objects shifted, but only if these objects served as potential reach targets. Our results suggest that the integration of allocentric information into the reach plan is determined by contextual factors, in particular by the extent of spatial cue changes and the task-relevance of allocentric cues.

General introduction

When performing reaching movements to a remembered visual target, the brain uses two classes of reference frames, the ego- and the allocentric reference frame (Battaglia-Mayer, Caminiti, Lacquianiti, & Zago, 2003; Colby, 1998). While a vast number of studies have shown that egocentric reference frames are used to plan and execute goal-directed reaching movements (e.g., Cohen & Anderson, 2002; Lacquaniti & Caminiti, 1998; Thompson & Henriques, 2011), an increasing number of studies also demonstrated an additional influence of allocentric information on reaching behavior, arguing for the integration of multiple reference frames (e.g., Byrne & Crawford, 2010; Diedrichsen, Werner, Schmidt, & Trommershäuser, 2004; Krigolson, Clark, Heath, & Binsted, 2007; Krigolson & Heath, 2004; Obhi & Goodale, 2005; Schütz, Henriques, & Fiehler, 2013, 2015), even in conditions when allocentric cues are covert and fall along a virtual line (Carrozzo, Stratta, McIntyre, & Lacquaniti, 2002). Byrne and Crawford (2010) further examined how egocentric and allocentric information are combined for reaching. They asked participants to reach for a remembered, visual target (light dot), which was either presented in isolation or was surrounded by four landmarks. After this display disappeared, only the landmarks reappeared but were shifted in space. Even though participants were instructed to ignore the landmarks (using solely egocentric information), averaged reaching endpoints deviated systematically in the direction of the landmark shift. This result shows that allocentric information was integrated into the reach plan and influenced reaching endpoints. 

Overall, the scope of the previous studies is rather limited. First, these studies mainly investigated if allocentric information (in addition to egocentric information) contributes at all to reaching behavior but hardly examined how allocentric information is used if multiple cues are available in the environment. Second, previous studies presented rather isolated and abstract stimuli on a blank screen, which had little relevance for the task. Third, task instructions usually prompted participants to apply either an egocentric or an allocentric reference frame, biasing the participants' behavior in one direction. In this study, we aim to investigate whether and how different allocentric cues contribute to reaching movements depending on (a) the change and (b) the task relevance of allocentric cues in a naturalistic everyday scene. Therefore, we utilized a memory-guided reaching task that did not bias participants to use either an egocentric or an allocentric reference frame. 

A recent study (Fiehler, Wolf, Klinghammer, & Blohm, 2014) provided first answers to the question of which factors influence the integration of allocentric information for planning and executing reaching movements. In the experiment, participants freely explored an image of a breakfast scene presented on a computer screen, which contained multiple objects on a table (table objects [TOs]) and in the background (background objects [BOs]). After scene encoding and a short delay, a test scene reappeared for 1 s with one TO missing indicating the reach target. Moreover, one, three, or five of the remaining TOs or one BO was shifted either to the left or to the right. The test scene was followed by a blank screen asking participants to perform a memory-guided reaching movement toward the target position (i.e., the position of the missing object). The main finding was that reach endpoints deviated in the direction of the object shifts and scaled with the number of objects shifted, but only if TOs were shifted, whereas a shift of BOs led to no deviations of reach endpoints. The authors concluded that allocentric information contributes to reaching movements to targets in naturalistic scenes, but this contribution depends on the extent of changes in the scene. 

However, the study by Fiehler et al. (2014) cannot explain the lack of an effect when BOs were shifted. Finding an explanation for this observation could reveal factors that are important to understand how allocentric information is integrated into the movement plan. The goal of the present study was to test two possible explanations. First, we examined if a change of allocentric information in a scene must exceed a certain extent in order to influence reaching behavior. Given the fact that a shift of at least three TOs was necessary to produce a significant deviation of reach endpoints in the previous study (Fiehler et al., 2014), it is conceivable that one shifted BO was not sufficient to cause any behavioral effect. Second, we investigated if only allocentric information relevant for the task contributes to reaching movements. In the previous study (Fiehler et al., 2014), TOs served as potential reach targets and thus were highly relevant to successfully perform the task, whereas BOs never served as reach targets. To this end, we performed two experiments in which we manipulated the number of shifted objects (Experiment 1) and the task relevance of shifted objects (Experiment 2) in the naturalistic scene. 

Experiment 1

Introduction

In order to test whether a certain amount of allocentric cue change in a scene is necessary to influence reach behavior, we applied a similar task as in the study by Fiehler et al. (2014), but this time we systematically varied the number of shifted BOs (instead of TOs). To this end, we added BOs and shifted either one, three, or five BOs left- or rightwards, comparable to the TO shifts applied in the previous study. Again, TOs served as potential reach targets. If a minimum number of shifted objects in the scene is needed to cause a substantial effect on reach endpoints, we expect an increase in the deviation of reach endpoints in the direction of the BO shifts, as was found for TO shifts before (Fiehler et al., 2014). Moreover, the amount of reach endpoint displacements should be similar for shifts of BOs and TOs. 

Methods

Participants

We recorded data from 15 subjects. Two participants were rejected due to fixation behavior (<50 % trials with correct fixation) and one subject was rejected because 63% of trials were classified as unsure response or no response was given. Thus, the final sample consisted of 12 participants (five female) with normal or corrected-to-normal vision. Participants were right-handed as assessed by the Edinburgh handedness inventory (Oldfield, 1971; M = 95.73 ± 12.4) and ranged in age from 20 to 31 years (M = 24.4 ± 2.9 years). They received course credit or were paid for their participation. The experiment was conducted in agreement with the ethical guidelines of the local ethics committee of the University of Giessen in compliance with the Declaration of Helsinki (2008). 

Materials

Stimuli consisted of 3-D–rendered images of a breakfast scene. Images were created in SketchUp Make 2013 (Trimble Navigation Ltd., Sunnyvale, CA) and afterwards rendered with Indigo Renderer 3.6.26 (Glare Technologies Ltd.) with a resolution of 3562 × 2671 pixels. The breakfast scene contained six TOs consisting of a coffee mug, a plate, an espresso cooker, a Vegemite jar, a butter dish, and an egg cup on a brown table that were placed 90 cm in front of a gray wall. Furthermore, five BOs, consisting of a table, a plant, a chair, a floor lamp, and a painting on the wall, were located in the surrounding area. Objects were taken from the open access online 3-D gallery of SketchUp. Object properties are summarized in Table 1. We set the six TOs in 40 different arrangements whereas the BOs always appeared at the same position (encoding image). The TOs in any arrangement were placed at one of three possible horizontal depth lines that were equally spaced (19.5 cm starting from the front table edge) on the table following three criteria: (a) minimum of one and maximum of three objects were positioned at every depth line, (b) objects were placed with a distance to the edges of the table so that in case of object displacement, it never stood right at the table edge or in the air next to the table, and (c) <20 % of an object was occluded by another object. Based on the encoding images, we created test images, in which one of the TOs was missing (reach target). In seven out of nine of the test images, other objects (TOs and/or BOs) in the scene were shifted horizontally between 3.08° and 4.08° (M = 3.61° ± 0.37°) either to the left or to the right (50% leftward displacement). Variations in the horizontal displacement arose from the fact that objects were placed at different depth lines relative to the virtual camera position. Hence, similar physical shifts of objects at different depth lines in 3-D space would result in different displacements in the 2-D image. In the remaining two out of nine of the test images, no objects were shifted. These images served as control condition. 

Table 1

 

Maximum height, width, and distance to camera of objects in the scene in cm, based on the actual properties in SketchUp. Notes: Table objects had no fixed distance to the camera as they were randomly placed on one of three different depth lines. However, the reported size relates to their absolute values in SketchUp. Some background objects were not fully visible due to an overlap with other background objects or partial cutting by the image borders. In that case, the absolute size of the actually visible object part is reported here.

 

In total, 1,120 images were created, including 40 encoding images, 840 test images (120 with TO shifts, 720 with BO or BO-and-TO shifts), and 240 control images. Furthermore, from each of the 40 encoding images, a scrambled version of the image made up of 768 randomly arranged squares was created and used for masking of the encoding image. 

Apparatus

Stimuli were presented on a 19-in. (40.5 × 30 cm) CRT monitor (Ilyama MA203DT) with a resolution of 1280 × 960 pixels and a refresh rate of 85 Hz. To reduce the influence of a high-contrast frame around the scene, a black cardboard frame (70 cm × 50 cm) was attached to the monitor. Participants sat at a table with their heads stabilized on a chin rest with a distance of roughly 47 cm from the eyes to the center of the screen. A decimal keyboard was placed in front of the subjects with the start button aligned to the chin rest and the center of the screen with a distance of 24 cm from the screen. Reaches were performed with the right index finger and recorded with an Optotrak Certus (NDI, Waterloo, Ontario, Canada) tracking system with a sampling rate of 150 Hz using one infrared marker attached to the fingertip of the right index finger. To control for correct fixation behavior, eye movements were recorded using an EyeLink II system (SR Research, Osgoode, Ontario, Canada) with a sampling rate of 500 Hz. To run the experiment and to control the devices we used Presentation 16.5 (Neurobehavioral Systems, Inc., Berkeley, CA). 

Procedure

The procedure of an example trial is depicted in Figure 1. At the onset of every trial, a fixation cross on a gray screen appeared, prompting participants to fixate and press a button in order to perform a drift correction for the EyeLink II. Thereafter, the encoding image of the breakfast scene was presented. Subjects freely explored the scene without any time constraints and terminated the encoding phase by pressing the start button. Then a scrambled version of the encoding scene appeared for 200 ms to avoid afterimages, followed by a delay phase of 1800 ms with a gray screen and a central fixation cross. Subjects were instructed to fixate the cross until the end of the reaching movement in order to provide a stable egocentric reference and to reduce the intersubject variability of reach endpoints due to gaze shifts during the delay period (cf. Byrne & Crawford, 2010). After the delay, the test image, which lacked one TO, was presented for 1000 ms. The trial continued with a short tone, which signaled the participants to perform the reaching movement toward the remembered location of the target object onto a gray screen. Thus, reaches were performed with gaze kept on the fixation cross and without any visual information of the encoding or test images. In this way we ensured that allocentric information could not be used for subsequent online corrections during the reaching movement, which would have led to an allocentric bias. 

Figure 1

 

Trial scheme of one example trial (control condition). (A) First, the encoding image was presented and subjects terminated the exploration of the image by button press. (B) Then, a scrambled version of the encoding image was presented for 200 ms, followed by (C) a delay which lasted for 1800 ms. (D) Thereafter, the test image was presented for 1000 ms before (E) a tone prompted subjects to reach to the target onto a gray screen while fixating the cross at the center of the screen.

Participants were instructed to reach to the location of the missing object as accurately as possible; we did not instruct them to reach to a specific part of an object. However, reach endpoints in the control condition were very consistent irrespective of the target object width (SD of reach endpoints, x-axis: 1.38–1.64 cm, y-axis: 0.29–0.30 cm), suggesting a uniform reaching behavior across all objects. Whenever they were unsure about the target location or identity, they had to reach to a marked location at the lower right edge of the monitor. These invalid trials were repeated at the end of the experiment. If subjects released the button before the go signal, they received feedback and these invalid trials were also repeated at the end of the experiment. 

Participants performed five experimental conditions (for examples, see Figure 2). In all experimental conditions, one of six TOs was always removed from the test image, which served as the reach target. In the TO-5 condition, the remaining five TOs were shifted either to the left or to the right. In the BO conditions, one, three, or all five BOs were shifted left- or rightward (BO-1, BO-3, BO-5). In the control condition, all objects remained stationary. In all conditions, left- and rightward object shifts were balanced with 50% of trials in each direction. There were two additional conditions in which the BO and the TO were shifted in the same or in the opposite direction. These conditions, however, are not relevant for answering the research question of this article and will be not presented here. 

Figure 2

 

Examples of encoding and test images for different conditions. Object names in the box indicate the reach target (missing object on the table).

Each participant completed a minimum of 1,320 trials. Because some trials were repeated (criteria see above), the actual number of performed trials varied from 1,322 to 1,435 trials. Trials were separated in five sessions with one session per day that lasted about 1 hr with one break in between. Trials were presented in pseudorandomized order with a random sequence of conditions and encoding images within a session but fixed trial combinations between sessions. A trial was never followed by a trial containing the same encoding image. 

Every test image was presented once (except the repeated trials due to unsure or too early responses, see above), apart from test images of the control condition, which were presented twice because of their importance for calculating reaching errors (see Data reduction). 

Data reduction and statistical analysis

Data preprocessing was performed with MATLAB R2007b (The MathWorks, Inc., Natick, MA) and inferential statistics with R 3.1 (R Development Core Team, www.r-project.org). All statistical tests were computed with an alpha level of 0.05. If correction for multiple testing was necessary, Bonferroni-Holm correction was applied. In case the assumption of sphericity for an ANOVA was violated (tested with Mauchly's sphericity test), Greenhouse-Geisser correction was applied. 

First, we inspected the eye tracking data and discarded trials from further data analysis in which subjects' gaze deviated more than 2.5° from the center of the fixation cross during a period beginning from delay onset till the end of the reaching movement. All in all 597 trials (3.76%) were rejected due to bad fixation. Second, reaching onsets and offsets were defined for every trial. The moment subjects released the response button determined the reaching onset. Reach offsets were calculated from Optotrak data and defined as the first time point during the movement when velocity dropped below 20 mm/s if the index finger reached a maximum distance of about 3 cm from the screen. Reach endpoints were extracted at the time of reach offset. Some trials were excluded because reaching offsets or endpoints could not be extracted due to rarely occurring interferences of the infrared markers of the Optotrak with the IREDs of the EyeLink II (198 trials = 1.3%). Third, we excluded trials in which reaching endpoints deviated more than ±2.5 SD in horizontal or vertical direction from the group mean in each condition for each object shift direction (291 trials = 1.93%). Taken together, from originally 15,840 trials of all participants, 14,755 valid trials (93.14%) remained. From these trials, 10,828 trials (93.99% of 11,520 trials) belonged to the conditions of interest (control, TO-5, BO-1, BO-3, BO-5) and were entered into further analysis. Reported results refer only to these conditions. 

To investigate the influence of object shifts (i.e., allocentric information) on reaching endpoints, we calculated allocentric weights for every subject and every condition by linear regression fit. First, we determined reaching errors as the horizontal distance of the reach endpoint and the actual target position of the encoding image. Therefore, we averaged reach endpoints of the control condition of all subjects for every combination of object arrangements and target objects separately. Since none of the objects were displaced in the control condition, no systematic reaching errors should have occurred. These averaged reach endpoints were used as actual target positions. Then, we calculated the differences between reaching endpoints of the other experimental conditions from the corresponding target position in the horizontal plane. This resulted in positive values for misestimations to the right and negative values for misestimations to the left. Moreover, we determined maximal expected reaching errors (MERE) for every image after an object shift by assuming that subjects completely relied on the allocentric information of shifted objects and thus produced reaching endpoints equal to the amount of the objects' displacement. To this end, we averaged the amount of displacement of the shifted objects for every image. If only one object was shifted, only this was taken into account. In case that more than one object were shifted, the different displacements were averaged. Please note that the perceived object shift depended on the object's location in depth. For the regression fit, the MERE was used as a predictor and the actual reaching error as a dependent variable for the two shift directions within one condition for every subject. The resulting slope of the regression line indicated the extent to which a participant relied on the allocentric information of object displacements and thus was used as allocentric weight for further analysis. 

We performed two-sided one-sampled t tests to investigate whether group allocentric weights for the different conditions differed significantly from zero. Since reaching errors and thus allocentric weights were computed on the basis of the results of the control condition, a test of weights against zero corresponds to a statistical comparison to the control condition. In order to assess the impact of allocentric information of BO on reaching endpoints, we conducted a one-way repeated-measures ANOVA for conditions with BO shifts with the factor number of shifted BO and three levels (BO-1, BO-3, BO-5). In case of significant main effects, we conducted two-sided post hoc t tests for paired samples. 

In order to investigate a potential influence of selective overt visual attention for task-relevant objects on the observed pattern of reach endpoints, we computed a heatmap of fixation densities of the encoding phase. To this end, we calculated the mean fixation point for every fixation starting from the second fixation until the end of the encoding phase for each subject in all conditions. Then, we collapsed the mean fixation points in one heatmap to generate an overview about subjects' average fixation behavior. For a better comparison of the fixation behavior in Experiments 1 and 2, we present the heatmaps of both experiments together in the Results section of Experiment 2 (see Figure 8). 

Results

As illustrated in Figure 3, reaching errors in condition TO-5 deviated in the direction of TO shifts, whereas no systematic reaching errors occurred when one (BO-1), three (BO-3), or five (BO-5) BOs were shifted. Figure 4A depicts the actual reaching errors and the corresponding MERE of one prototypical participant for the conditions TO-5 and BO-5 for leftward (negative values) and rightward (positive values) object displacements. The slope of the regression line defined the allocentric weight of the respective condition. Allocentric weights of TO shifts significantly differed from zero, t(11) = 7.075, p < 0.001, and were substantially higher compared to BO shifts irrespective of the number of shifted objects (Figure 4B; Table 2). 

Figure 3

 

Mean horizontal and vertical reaching errors (in cm) of every subject for each condition. Leftward object shifts are depicted in white, rightward objects shifts are in gray.

Figure 4

 

(A) Example reaching errors of one subject for the condition with shifts of five TOs (TO-5) and five BOs (BO-5). Actual reaching errors are plotted against the MERE. Moreover, the linear fit (slope = allocentric weight) is depicted. (B) Allocentric weights for every condition averaged over all subjects. Error bars indicate 1 SEM.

Table 2

 

Summary of allocentric weights for all conditions. Notes: Range, mean, and standard deviation of the sample are listed. Results of two-sided one-sampled t tests are Bonferroni-Holm corrected. * indicates significant results.

 

To assess the effect of an increasing number of BO shifted on reaching endpoints, we conducted a one-way repeated-measures ANOVA. We found a main effect for the number of shifted BOs, F(2, 22) = 6.121, p = 0.008. Post hoc t tests indicated that allocentric weights of BO-1 were smaller than those of BO-3, t(11) = −2.913, p = 0.042, and of BO-5, t(11) = −2.895, p = 0.042, whereas allocentric weights of BO-3 and BO-5 did not differ, t(11) = −1.495, p = 0.163. However, the effect size was very small (η² = 0.11) indicating a rather weak influence of BO shifts on reaching endpoints. 

Discussion

In this experiment, we tested whether a decreased number of BOs shifted in the scene (one BO versus three or five TOs) could account for the lacking influence of BO shifts on reaching endpoints in the study by Fiehler et al. (2014). Therefore, we manipulated the allocentric information by shifting one, three, or five BOs left- or rightwards after scene encoding and before participants performed a reaching movement to the remembered target position. No visual information about the scene was given during the reaching movement and gaze and body position remained fixed. If a minimum extent of spatial change of allocentric cues is necessary to influence reaching movements, we predicted a similar increase of mean allocentric weights with the number of shifted BOs in Experiment 1 as was observed previously in conditions with increasing TO shifts (Fiehler et al., 2014). 

We found no influence of BO shifts on reaching endpoints irrespective of the number of shifted BOs. In all experimental conditions including BO shifts, averaged allocentric weights did not differ from zero. Nevertheless, allocentric weights of the condition with one shifted BO were smaller than conditions with three or five shifted BOs. Thus, there was a small increase in displacements of reaching endpoints with an increasing number of shifted BOs. However, as the effect size of this result was very small and allocentric weights of these conditions did not differ from zero, this influence is negligible. This becomes even more evident if we compare the allocentric weights observed here for three (0.02) and five (0.04) BO shifts with the allocentric weights of three (0.1) and five (0.44) TO shifts of the previous study (Fiehler et al., 2014) or five TO shifts (0.47) assessed in this experiment (compare to Figure 7B below). In sum, the lack of an effect of allocentric information of BO stimuli on reaching endpoints can't be explained by a decreased number of shifted BOs. 

The results of Experiment 1 further demonstrated that shifts of five TOs led to a displacement of reaching endpoints into the direction of the object shift. This supports the results of the previous study (Fiehler et al., 2014) and confirms that the slight adjustments of the procedure and the use of 3-D-rendered scenes instead of photographic images did not significantly influence the present results. 

Experiment 2

Introduction

A second possible explanation for the absence of an effect in conditions with BO shifts in the study by Fiehler et al. (2014) is the fact that BOs never served as potential reach targets and hence, were not relevant to perform the task. Thus, participants may have paid no attention to these objects and in turn did not integrate this information into the reach plan. To address this hypothesis, we conducted a second experiment that was very similar to the paradigm that we used in our first experiment. The crucial difference was that BOs instead of TOs served as reach targets. We applied the same procedure but slightly adjusted the stimuli to match the task requirements (see Methods for further details). If the task relevance of an object is essential for its use as allocentric cue for reaching, we expect systematic reaching errors only in conditions with BO shifts. Furthermore, these errors should increase with the number of shifted BOs and should reveal a pattern similar to the results for TO shifts found by Fiehler et al. (2014). In contrast, we expect no systematic reaching errors in conditions with TO shifts in Experiment 2 as these objects are not relevant to perform the task. 

Methods

Participants

We recorded data from 11 participants. Two subjects were rejected due to bad fixation behavior (<50 % trials with correct fixation). Thus, data of nine participants (six female) with normal or corrected-to-normal vision was further analyzed. Participants were right-handed as assessed by the Edinburgh handedness inventory (Oldfield, 1971; M = 74.67 ± 26.91) and ranged in age from 19 to 27 years (M = 22.11 ± 3.14 years). They received course credit or were paid for their participation. The experiment was conducted in agreement with the ethical guidelines of the local ethics committee of the University of Giessen in compliance with the Declaration of Helsinki. 

Materials

Stimuli were created as described for Experiment 1. The breakfast scenes of Experiment 2 differed from the scenes used in Experiment 1 in the following ways. First, instead of six, only five TOs were placed on the table (the egg was omitted) in order to create a comparable nontarget stimulus set to Experiment 1 (five nontarget objects). Accordingly, we also increased the number of BO from five to six to create a comparable target-stimulus set to Experiment 1 (six target objects). In addition, we replaced the BO by six other objects (chair, vase, painting, calendar, clock, and ceiling lamp) that were more equal in size and generally smaller in width and thus should reduce the variance of reaching endpoints across the target objects. The object properties of the BOs used in Experiment 2 are summarized in Table 3. 

Table 3

 

Maximum height, width, and distance to camera of new background objects in the scene in cm, based on the actual properties in SketchUp. Notes: Painting, calendar, and clock had no fixed distance to the camera because their position altered on three different height levels on the wall. Some background objects were sometimes not fully visible due to an overlap with other objects. Therefore visible heights may vary from the actual height depending on the used object arrangement.

 

We created 14 different object arrangements. Placements of TO followed the same criterion as in Experiment 1. The BO were positioned (a) with a distance to the edges of the image so that in case of object displacements they would be still completely visible, and (b) that they were never occluded >20 % by another object even after object displacement. The painting, calendar, and clock were placed at three different heights with (a) minimum of one object placed at every height level, and (b) the calendar never placed on the highest level in order to minimize unrealistic object arrangements in the scene. The distance of the low height level from the ground was 107.55 cm, of the middle level 125.99 cm, and of the high level 144.43 cm. Distances from the height levels to the camera were 278.97, 279.51, and 281.13 cm for the low, middle, and high level, respectively. The positions of the other BOs were fixed on one horizontal line for each object in different distances to the camera (see Table 3). Again, we created test images, but this time with one BO (instead of one TO) missing, which served as reach target. In two out of three of the test images, all five TOs or one, three, or five BOs were displaced either to the left or to the right (50% leftward displacements). Amounts of horizontal displacement varied between 3.56° and 4.47° (M = 3.86° ± 0.33°). In the remaining one out of three test images, no objects were shifted (control condition). 

In total, 266 images were rendered, including 14 encoding images, 168 test images (42 with TO shifts, 126 with BO shifts), and 84 control images. Moreover, from each of the 14 encoding images, a scrambled version made up of 768 randomly arranged squares of the image was created and used to mask the encoding image. 

Apparatus and procedure

The same set-up and trial procedure were used as in Experiment 1. Participants performed five experimental conditions (for examples see Figure 5). In contrast to Experiment 1, one of the six BOs was always removed from the test image and served as reach target. Each participant completed a minimum of 504 trials. Trials in which participants responded too early, did not respond, or were unsure about the target position were repeated at the end of the experiment. Thus, the number of actually performed trials varied from 504 to 524 trials. Trials were presented in pseudorandomized order, but a trial was never followed by a trial containing the same encoding image. The experiment was conducted in two sessions with all trials in the first and a repetition of every trial in the second session. Sessions were performed on two different days with one break in between. 

Figure 5

 

Examples of encoding and test images for different conditions. Object names in the box indicate the reach target (missing object in the background).

Data reduction and statistical analysis

Data preprocessing was performed with MATLAB R2007b (The MathWorks Inc.) and inferential statistics with R 3.1 (R Development Core Team). All statistical tests were computed with an alpha level of 0.05. If correction for multiple testing was necessary, correction following Bonferroni-Holm was applied. In case the assumption of sphericity for an ANOVA was violated (tested with Mauchly's sphericity test), Greenhouse-Geisser correction was applied. 

As in Experiment 1, first eye-tracking data was inspected in order to detect incorrect fixations (deviations >2.5° from fixation cross) and correspondent trials were discarded from further analysis. This applied to 479 trials (10.56%). Next, reaching onsets and offsets were defined following the same criteria as in Experiment 1. Two trials (0.05%) had to be rejected from further analysis because reaching endpoints could not be determined. Finally, we excluded trials in which reaching endpoints deviated more than ±2.5 SD in horizontal or vertical direction from the group mean in each condition for every object shift direction (106 trials = 2.61%). From originally 4,536 trials, 3,948 valid trials (87.04%) were entered into statistical analysis. 

To investigate the influence of object shifts on reaching endpoints, we calculated allocentric weights as described for Experiment 1. We performed two-sided one-sampled t tests to investigate whether group allocentric weights for every condition differed significantly from zero. In order to assess the impact of allocentric information by BO on reaching endpoints, we conducted a one-way repeated-measures ANOVA for conditions with BO shifts with the factor number of shifted BO and three levels (BO-1, BO-3, BO-5). 

Again, we computed a heatmap of fixation densities for the encoding phase as in Experiment 1. While in Experiment 1 target objects were placed in the lower part of the scene, in Experiment 2 they were located in the upper part of the scene around the future location of the fixation cross. Thus, subjects tended to fixate this region more often as in Experiment 1, even though objects were never placed in this area. To take this into account, we excluded fixations that fell into the area of the fixation cross from computing the heatmap. 

Results

Figure 6 depicts horizontal and vertical reaching errors for every condition. Right- and leftward BO shifts led to systematic reach errors most pronounced for five object shifts (BO-5), smaller for three shifted objects (BO-3), and absent for one object shift (BO-1). In contrast to Experiment 1, we found no systematic reach errors if five TOs (TO-5) were shifted. Accordingly, allocentric weights differed significantly from zero in BO-5 and BO-3 but not in BO-1 and TO-5 (Table 4). 

Figure 6

 

Mean horizontal and vertical reaching errors (in cm) of every subject for each condition. Leftward object shifts are depicted in white, rightward objects shifts are in gray.

Figure 7

 

(A) Mean allocentric weights for the TO condition and all BO conditions. Error bars represent 1 SEM. (B) Mean allocentric weights reported by Fiehler et al. (2014). Instead of BOs, TOs served as reach targets. In TO conditions, one, three, or five TOs were shifted either to the right or to the left. In BO conditions (BO-1_T, BO-1_C, BO-1_P), only one of the three BOs was displaced.

Figure 8

 

Heatmaps of fixation densities for the encoding phase averaged across all subjects and conditions for (A) Experiment 1 and (B) Experiment 2. The high density of fixations at the center of the image corresponds to the location of the upcoming fixation cross after image encoding (cf. trial schedule in Figure 1).

Table 4

 

Summary of allocentric weights for all conditions. Notes: Range, mean, and standard deviation of the subject group are listed. Results of two-sided one-sampled t tests are Bonferroni-Holm corrected. * indicates significant results.

 

We performed a one-way repeated-measure ANOVA for the BO conditions on allocentric weights to evaluate the influence of an increasing number of shifted BO on reaching endpoints. We found a main effect for the number of shifted BO, F(2, 16) = 7.272, p < 0.001, η² = 0.378. Post hoc t tests indicated that allocentric weights of BO-1 were significantly smaller than weights of BO-5, t(8) = −3.379, p = 0.029. Allocentric weights of BO-3 did not differ from weights of BO-1, t(8) = −2.124, p = 0.133, and weights of BO-5, t(8) = −2.032, p = 0.133. Figure 7A shows the mean allocentric weights for every condition. For comparison, we also depicted the mean allocentric weights published by Fiehler et al. (2014) in Figure 7B. 

For Experiments 1 and 2, we computed heatmaps of fixation densities over all subjects and conditions for the encoding phases. As illustrated in Figure 8, participants mainly fixated the objects on the table and hardly fixated the objects in the background in Experiment 1 (Figure 8A), whereas in Experiment 2 the reversed fixation pattern occurred (Figure 8B). Here, most of the fixations fell on objects in the background while nearly no fixation was kept on objects on the table. This result pattern clearly shows that participants fixate on objects that serve as potential reach targets and thus are relevant for the task (i.e., TOs in Experiment 1 and BOs in Experiment 2). 

Discussion

In Experiment 2, we tested the hypothesis whether task-relevant allocentric cues only are integrated into the reach plan and thus influence reaching behavior. In contrast to Experiment 1, BOs (instead of TOs) served as reach targets and thus became relevant to perform the task. We predicted a similar pattern of mean allocentric weights for BO shifts as was observed for TO shifts in the previous study (Fiehler et al., 2014). In line with our prediction, we observed systematic endpoint errors in the direction of BO shifts. Moreover, these endpoint error deviations increased with the number of shifted BOs. This was confirmed by averaged allocentric weights, which revealed a similar increase compared to the allocentric weights in the study of Fiehler et al. (2014). In contrast to Experiment 1, we found no effect on reach endpoints for conditions with TO shifts despite the fact that all five TOs were shifted. Therefore, we conclude that the task relevance of an object determines whether an object is used as an allocentric cue or not. 

General discussion

There is converging evidence that reach targets are represented in both egocentric (e.g., Cohen & Anderson, 2002; Lacquaniti & Caminiti, 1998; Thompson & Henriques, 2011) and allocentric reference frames (e.g., Byrne & Crawford, 2010; Diedrichsen et al., 2004; Krigolson et al., 2007; Obhi & Goodale, 2005; Schütz et al., 2013, 2015). So far, little is known about the factors contributing to how allocentric information is used for reaching when multiple environmental cues are available in more complex situations, such as in naturalistic images. Here, we followed the approach of Fiehler et al. (2014) and tested two hypotheses derived from their study. First, we examined whether the spatial change of allocentric cues in a scene must exceed a certain extent to influence reach behavior. Second, we tested if task relevance of allocentric cues is an important factor for determining whether or not they are used for reaching. In our first experiment, we shifted one, three, or five BOs or five TOs while TOs served as potential reach targets. Here, we replicated the findings of Fiehler et al. (2014) and found a systematic shift of reach endpoints in the direction of the TO shifts. However, an increasing number of shifted BOs did not influence reach endpoints, even if all available BOs were shifted. In our second experiment, again one, three, or five BOs or five TOs were shifted; but this time BOs served as potential reach targets. Here, we found the reversed effect of Experiment 1 (i.e., reaching errors varied systematically with the number and direction of shifted BOs while being unaffected by shifts of TOs). Based on these findings, we conclude that task relevance is a crucial factor for the use of allocentric cues for reaching. 

If task relevance is given for a group of objects, we observed that the number of shifted objects determines their influence on reaching endpoints. We found allocentric weights increasing from 0% for one to 48% for five shifted objects, irrespective of whether task-relevant objects were placed in the background or on the table. This result is comparable to the study by Fiehler et al. (2014) who also found a systematic increase of allocentric weights with an increasing number of shifted task-relevant TOs varying from 1% for one to 43% for five shifted objects. Accordingly, previous work on the Roelofs effect showed that the mislocalization of a target within a frame increases when the full frame is shifted compared to conditions in which only parts of the frame are being displaced (Walter & Dassonville, 2006). This indicates that an increasing amount of changing allocentric information has a cumulative impact on the perceived target localization, especially if it comprises all available allocentric cues. However, as we revealed in Experiments 1 and 2, this finding crucially depends on the condition that the group of misplaced objects is task-relevant. We suggest that both egocentric and allocentric information are integrated into the reach plan depending on bottom-up and top-down processes, namely the amount of changes in the scene and the task relevance of allocentric cues, respectively. 

Our findings also support results on visual attention and task relevance of objects in real world and natural scenes. It has been demonstrated that overt visual attention is mainly distributed to objects that are relevant to perform a task reflected in more fixations on task-relevant than -irrelevant objects (Ballard & Hayhoe, 2009; DeAngelus & Pelz, 2009; Land & Hayhoe, 2001) and longer fixation durations (Mills, Hollingworth, Van der Stigchel, Hoffman, & Dodd, 2011). In order to test whether participants in our experiments also showed an attentional preference for task-relevant objects during image encoding, we computed heatmaps illustrating fixation densities for the encoding images of all subjects (Figure 8). Figure 8 shows that participants fixated the area around the TOs more often than the area around the BOs in Experiment 1, while the reversed fixation pattern was visible in Experiment 2 (i.e., more fixations on background than on TOs). Consistent with previous findings on top-down control of eye movements in real world and natural scenes (Ballard & Hayhoe, 2009; DeAngelus & Pelz, 2009; Land & Hayhoe, 2001; Mills et al., 2011), fixations (i.e., overt visual attention) were predominantly shifted to objects that were relevant to perform the task. Moreover, the fixation behavior found here is in line with the study by Fiehler et al. (2014) where subjects mainly fixated the area around the task-relevant TOs. We conclude that overt visual attention is mainly distributed to areas with task-relevant allocentric cues that are consequently integrated into the reach plan. 

Previous research suggests that task relevance of objects in a scene can improve the detection of changes of object properties (Triesch, Ballard, Hayhoe, & Sullivan, 2003) and prioritize their retention in visual working memory (Maxcey-Richard & Hollingworth, 2013). In our task, a short delay was implemented between the encoding and the test image, which required maintaining object positions in visual working memory. This information was subsequently used to detect the target location and to perform the reaching movement. Subjects' fixation behavior (Figure 8) suggests that they prioritized the position of task-relevant objects during scene encoding which may have facilitated target detection and the selection of task-relevant allocentric cues for reaching. 

Besides task relevance, the distance between target and allocentric cues seems to influence how and to which extent these cues contribute to reaching movements (Camors, Jouffrais, Cottereau, & Durand, 2015; Diedrichsen et al., 2004; Krigolson et al., 2007). In our experiments task-relevant objects, which served as reach targets, were always placed in the direct vicinity of each other, excluding a potential impact of target-landmark distance. However, some (but not all) task-irrelevant objects were presented further away from the target, which might have led to an attenuated contribution of task-irrelevant objects on reaching. Nevertheless, we believe that a larger distance between task-irrelevant objects and target cannot fully account for the lack of an effect of irrelevant objects. It is rather likely that participants initially ignored task-irrelevant objects during image encoding as evidenced by the heatmaps in Figure 8 and thus did not integrate this allocentric information into the reach plan. 

Furthermore, a temporal delay between stimulus presentation and action performance can influence the weighting of egocentric and allocentric information with stronger weighting of allocentric information for delayed than immediate movements (Bridgeman, Peery, & Anand, 1997; Chen, Byrne, & Crawford, 2011; Hay & Redon, 2006; Obhi & Goodale, 2005). Importantly, allocentric information is also incorporated in immediate reaches as recently demonstrated by a study on the Roelofs effect (Taghizadeh & Gail, 2014) and as we demonstrated in our experiments. Since allocentric coding is supposed to be stronger for delayed than immediate reaches (Bridgeman et al., 1997; Chen et al., 2011; Hay & Redon, 2006; Obhi & Goodale, 2005), the allocentric weights we observed here could have even been higher with a temporal delay before the reach. 

The extent to which results from studies using photographs or 3-D-rendered scenes can be transferred to real world situations poses an interesting question for future research. It has been demonstrated that real-world objects were better recalled and recognized than photographs or line drawings of these objects (Snow, Skiba, Coleman, & Berryhill, 2014). Based on this finding, one could argue that the present results cannot be generalized to naturalistic behavior. However, we believe that our approach is an important intermediate step in order to transfer outcomes from laboratory settings to the real world. 

Overall, our findings extend the current scientific body showing that task relevance of allocentric cues determines their contribution to reaching movements if multiple cues are available in a more complex and naturalistic environment. Moreover, this influence on reaching behavior further depends on the extent of changes of task-relevant allocentric information. 

Acknowledgments

This research was funded by the International Research Training Group (IRTG) 1901 “The Brain in Action” by the German Research Foundation (DFG). Moreover, we would like to thank Lisa Wallner, Immo Schuetz, and Lena Klever for their support. 

Commercial relationships: none. 

Corresponding author: Katja Fiehler.