The role of memory in visually guided reaching

Abstract

People can be shown to use memorized location information to move their hand to a target location if no visual information is available. However, for several reasons, memorized information may be imprecise and inaccurate. Here, we study whether and to what extent humans use the remembered location of an object to plan reaching movements when the target is visible. Subjects sequentially picked up and moved two different virtual, “magnetic” target objects from a target region into a virtual trash bin with their index fingers. In one third of the trials, we perturbed the position of the second target by 1 cm while the finger was transporting the first target to the trash. Subjects never noticed this. Although the second target was visible in the periphery, subjects' movements were biased to its initial (remembered) position. The first part of subjects' movements was predictable from a weighted sum of the visible and remembered target positions. For high contrast targets, subjects initially weigh visual and remembered information about target position in an average ratio of 0.67 to 0.33. Over the course of the movement, weight given to memory decreased. Diminishing the contrast of the targets substantially increased the weight that subjects gave to the remembered location. Thus, even when peripheral visual information is available, humans use the remembered location of an object to plan goal-directed movements. In contrast to previous suggestions in the literature, our results indicate that absolute location is remembered quite well.

Introduction

People can move their hands to a previously seen target with closed eyes by using the target's memorized location; however, most hand movements are made under visual guidance. Visual information about a target object is typically available both for planning movements and for controlling them online. When performing complex tasks, humans appear to coordinate eye movements with hand movements to maximize the available visual information to guide hand movements and to minimize any reliance on memory (Ballard, Hayhoe, & Pelz, 1995). It has even been argued that only very little information is stored across saccades (Henderson & Hollingworth, 1999; Irwin, 1991) and that instead of relying on stored information, the “world is used as an external memory” (O'Regan, 1992; Rensink, 2000). When considered in the context of the spatial information needed to plan and guide hand movements, this might appear to make sense because remembered target location information is old and, in a changing world, possibly not correct anymore. Even when nothing has changed, memorized location information might be expected to be more uncertain than visual information, for example, because of noise introduced when memorized location is remapped to correct for eye movements (Henriques, Klier, Smith, Lowy, & Crawford, 1998) or is remapped relative to a more stable reference frame right away (Andersen, Essick, & Siegel, 1985; Sparks & Mays, 1983). These and other drawbacks could explain the lower precision of movements toward remembered than to visual locations (Binsted, Rolheiser, & Chua, 2006; Heath, Westwood, & Binsted, 2004). They might also explain the finding that subjects make repeated eye movements in natural eye–hand tasks (Ballard et al., 1995) or scene comparison tasks (Gajewski & Henderson, 2005) and that visual search may operate without the use of memory (Horowitz & Wolfe, 1998). 

That the brain relies largely on visual information for online control during hand movements is suggested by the fact that the eyes typically arrive at a target well before the hand does (e.g., Binsted, Chua, Helsen, & Elliott, 2001; Flanagan & Johansson, 2003; Neggers & Bekkering, 2000). By contrast, the relative start times of hand and eye movements are variable and depend on factors such as knowledge of the target location (Abrams, Meyer, & Kornblum, 1990; Pelz, Hayhoe, & Loeber, 2001). Often, the hand begins moving soon after the beginning of the eye movement to a target or even before (Aivar, Hayhoe, Chizk, & Mruczek, 2005; Binsted & Elliott, 1999; Carnahan & Marteniuk, 1991, 1994). This suggests that remembered location information could play a role in planning movements because positional acuity degrades with increasing eccentricity (Burbeck & Yap, 1990; Whitaker & Latham, 1997). 

We tested the hypothesis that humans use all available information about a target's location to plan hand movements to the target, including the visual information available in the peripheral visual field and memorized location information, but that the relative contributions of the different information sources depend on their relative reliability. When vision is significantly more reliable than the information stored in memory, vision will appear to dominate, but in conditions in which visual information is degraded, the brain will give more weight to the information in memory to plan movements. Such results would be in accordance with studies about perceptual cues that have shown that cue weighting depends on cue reliability (e.g., Alais & Burr, 2004; Ernst & Banks, 2002; Hillis, Watt, Landy, & Banks, 2004; Knill & Saunders, 2003; van Beers, Sittig, & Denier van der Gon, 1998). 

Methods

Overview

Subjects successively picked up two virtual, “magnetic” target objects from a target area using their right index fingers and put them in a virtual trash bin, which was located approximately 28 cm (≈33° of visual angle) to the left of the target area. In a third of the trials, the second target object shifted 1 cm up or down during the transportation of the first target object to the trash bin. This created a situation in which remembered location of the second target conflicted with its visually specified location prior to initiating the hand movement to pick it up. A brief flicker of the display at the onset of the first movement to the trash bin masked the perturbations so that subjects were unaware that they ever occurred. To estimate the weights that subjects gave to remembered and visually specified target location to plan the movement to the second object, we regressed subjects' finger locations at different points in time against the initial (remembered) position of the target object and its possibly new (visual) position. This revealed how the weight given to memory develops over the duration of the movement, with the weights at the beginning reflecting information used for planning and the weights at the end reflecting the (increasingly stronger) influence of online control. To confirm that only peripheral visual information was available for planning the movement to the second target and to estimate the time at which foveal information about the second object became available for online control, we recorded eye movements of four of our subjects. 

Subjects

Six female and four male subjects volunteered to participate in the experiment for payment, after being informed about what they were required to do. Most of them were students at the University of Rochester. They were all naive with respect to the purpose of the experiment. All subjects had normal or corrected-to-normal visual acuity and good stereo vision as tested by Randot ^® Stereotests (Stereo Optical Company Inc., Chicago, IL). This study is part of an ongoing project that had been approved of by the local ethics committee. 

Apparatus

Setup

Figure 1 is a schematic overview of the setup (A) and the working environment of the subject from his or her vantage point (B). Starting with their right index finger on a large starting cross, subjects were asked to pick up first the square “magnetic” target object and transport it to a “trash bin,” then pick up the round target object and put it in the trash as well. 

Figure 1

 

Schematic depiction of the experimental setup (A), an overview of a trial in which the second target is perturbed downward (B), and a schematic depiction of the possible locations of the targets relative to the trash bin (C). The gray lines along the circle indicate the possible locations of the targets.

Subjects rested their heads in a chin and head rest and viewed the experimental environment that was displayed on a screen through a mirror. The table underneath and the moving hand were obscured by the mirror. Subjects viewed the display stereoscopically through LCD shutterglasses. The starting cross, targets, and trash bin were made to appear on the table by rendering them in three dimensions at the appropriate locations and orientations. A virtual finger was rendered at the 3D position and orientation of the subject's real finger. To record the location and orientation of the finger over time, subjects wore a steel tube over their index finger with three infrared markers attached to it. These were tracked by an Optotrak 3020 system (NDI, Waterloo, Ontario, Canada) at 120 Hz. The information was used online to compute the 3D position and pose of the finger. We used linear extrapolation of the finger's position to correct for the approximately 20-ms delay between recording the finger's position and the appearance of the virtual finger in the display; thus, the virtual finger moved in real time with the actual, unseen finger. When visually compared with one's real finger by viewing the display through a half-silvered mirror, the only notable errors in the virtual finger appeared at the very beginnings and ends of movements, when accelerations and decelerations were high. 

On the table were two steel plates: one on the right and one on the left. The targets were projected on the right plate, and the starting cross and the trash bin were projected on the left plate. The plates were connected to a 5-V source and to a Northern Digital Optotrak Data Acquisition Unit II that recorded the voltage across each plate at 120 Hz. The steel tube worn over the index finger acted as a ground so that by measuring the voltage of the plates, we acquired precise measurements of the time that the finger left and arrived at the left or the right plate. These were used to determine the timing of the beginnings and ends of the submovements involved in the task. 

Calibration

Each time subjects came into the laboratory, we calibrated the virtual environment by measuring the positions of the eyes in space, the position and orientation of the display in space, and the position and orientation of subjects' fingers relative to the Optotrak markers. In a first calibration procedure, subjects matched the perceived location of a rendered dot with an Optotrak marker that they could see after removing the backing of the half-silvered mirror. This was done for 13 locations in two different depth planes in separate sequences for the right and left eyes. Using a least squares procedure, we used these data to fit both the positions of a subject's two eyes and the position and orientation of the virtual display (the image of the display screen behind the mirror). Afterward, the subjects checked whether the calibration was good by moving the Optotrak marker and see whether a rendered dot followed it accurately. In a second calibration procedure, we determined the position and orientation of subjects' index fingers and fingertips relative to the Optotrak markers. Subjects matched the perceived location of a rendered finger with their own index finger wearing the steel tube, which they could see through the half-silvered mirror. Then, the backing was replaced behind the mirror and a square target appeared. The subjects were asked to touch the target with their finger, of which the virtual image was visible, in such a way that they felt they were on the center of the target. This was repeated for five other target locations. The results were used to determine the vector difference between the Optotrak markers on the steel tube and where subjects perceived their fingertips to be. 

Electrooculogram

We measured the horizontal component of eye movements for four of our subjects by recording electrooculogram (EOG) signals. Two surface electrodes were pasted close to the outer canthi of both eyes. One electrode was pasted acentric on the forehead as a ground. Custom-made equipment recorded the potential differences between the electrodes at a rate of 1000 Hz. At the start of each new block of trials, these subjects went through a calibration procedure that entailed fixating the center of a cross. The cross was first presented at the center of the screen, and then it appeared at 15 locations equally dispersed along the horizontal midline of the screen, going from the left to the right. Afterward, a cross was presented, which moved with the eyes to check whether the calibration was reasonable. If, during the block, the recorded position of the eyes had drifted outside of the region in which it could be measured, we suspended the experiment and repeated the calibration procedure. 

Stimuli

Object appearance

The targets were a 7-mm-wide square and a circle with a diameter of 7 mm (≈0.8° of visual angle); the trash bin was a plus sign with a diameter of 12 mm (≈1.4°). The starting cross, trash, and target objects were all displayed in red on a dark background (a setting of 100% black; 0.03 cd/m ² as measured with a Minolta luminance meter LS-110, 0.00 cd/m ² as measured through the shutterglasses). The targets could be of high contrast (a setting of 60% red; 1.12 cd/m ², 0.14 cd/m ² through the shutterglasses) or low contrast (a setting of 30% red; 0.23 cd/m ², 0.03 cd/m ² through the shutterglasses). 

Object locations

The starting cross was always located at the center of the display's edge closest to the subject. Figure 1C schematically shows the possible locations of the targets and trash bin. The trash bin was located at the left side of an imaginary circle with a diameter of 28 cm (≈33°) and centered on the display. The two targets were located along the right edge of this circle (with a bit of scatter; the maximal distance between a target's center and the circle was 3 mm). The average locations of the two targets were selected to be 20° up or down with respect to the horizontal line between the center of the display and the center of the target space. On each trial, the positions of the two targets were randomly chosen from positions along the circle's edge that were within ±10° up or down from their average positions. Which target appeared above or below the other was randomly chosen from trial to trial. With this layout, the minimal possible distance between the two targets was 20° and the maximal possible distance was 60° with respect to the line between the center of the display and the center of the target space. This means that the minimal distance between the two targets was approximately 5.5° of visual angle and that the maximal distance was 17.5°. The whole constellation of trash bin and target areas was rotated around the center of the display by a random amount between 2.5° up and 2.5° down to have some variation in the location of the trash bin while keeping the relative locations of the target areas to the trash bin constant. 

Procedure

Overview of a trial

The room in which the experiment was performed was darkened. A subject started each trial by positioning the right index finger on the starting cross. After 500 ms, the cross spun, signaling the subject to start moving to the first (square) target. If the movement started before 100 ms after the start signal, a “too early” warning appeared and the trial was repeated later. On touching the target, it “magnetically” stuck to the finger, as indicated by the target increasing in size. The criterion for touching the target was that the fingertip had to be within 10 mm of the center of the target. At the moment that the subject's finger left the table to put the first target to the trash bin, we presented a 200-ms masking flicker, rendering the whole screen successively black for two frames and white for two frames. On two thirds of trials, nothing changed during the flicker, but on one third of trials, the second target shifted 1 cm up or down (perpendicular to the line between the trash bin and the center of target space). The subject put the target into the trash bin where it disappeared. If the finger was within 2.5 mm of the center of the trash bin, the subject was rewarded by an exploding trash bin—it tripled briefly in size after contact. This was done to focus subjects' attention on placing the objects in the trash bin, maximizing the chance that the subjects would fixate the trash bin area at least until the first object had been successfully been disposed of. Next, the subject moved the finger from the trash bin to the second (round) target. When the subject lifted his or her finger from the table, another 200-ms flicker was presented. The aim of this flicker was to make sure that the first portion of the movement only reflects information used in the planning phase and could not be adjusted on the basis of newly acquired foveal target location information. (As shown in the Results section, this would probably not have been necessary as many subjects appear to move their finger relatively long before the gaze arrives on the target anyway.) Subjects then picked up the second object and moved it to the trash bin. The display was flickered for 200 ms at the onset of this final movement to maintain consistency with the previous stages of the task. The trial ended after the subject had put the second target in the trash bin as well. 

Subjects were asked to move fast and precisely. If the complete trial was not finished within 6,000 ms, a message appeared, stating that it was too slow, and the trial was repeated later. 

Overview of the complete experiment

All subjects started the experiment with a practice block of 30 trials, which contained 15 unperturbed trials each of the high- and low-contrast target conditions. These data were not analyzed. For each of the two levels of contrast, each experimental block contained 30 unperturbed trials, 8 trials in which the second target shifted upward, and 8 trials in which the target shifted downward. Thus, there were 92 trials per block. Within each block, the trials were presented in random order. Nine subjects performed 12 blocks, and one subject performed 11 blocks of experimental trials. At the end of each block, a score was displayed, which corresponded to the number of trials on which subjects had precisely placed objects in the trash bin, that is, on which the trash bin had “exploded.” Subjects took a brief rest between blocks. They usually completed three blocks in one session and came back on other days for the other blocks. After the complete experiment, subjects were first asked whether they had noticed anything strange in the display and, then, whether they had noticed the second target shifting up or down. 

Data analysis

Finger

We used the signals from the contact of the finger with the steel plates to determine when the finger left or arrived at the starting cross, the targets, and the trash bin. We determined the constant pointing error relative to the center of the second target at the time that the subject arrived there, in both the horizontal and vertical directions. Negative values for the horizontal and vertical pointing error indicate hitting a location to the left and below the center of the target, respectively; positive values indicate a location to the right and above the target's center. Variable pointing error was computed by taking the standard deviation of the pointing errors for each subject and each condition separately, for both the horizontal and vertical directions. 

We analyzed the movement from the trash bin to the second target to quantify the weight that subjects assigned to memorized location and how this developed over time. Multiple linear regressions were performed with the vertical finger position as the dependent variable and the memorized and visual target location as independent variables for different time steps. The underlying model for the regression at each time step was that the vertical position of the finger could be given by the following linear function:  

$Y_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{r}}=W_{\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{y}}\times Y_{\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}+W_{\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}\times Y_{\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{u}\mathrm{a}\mathrm{l}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}+\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},$

(1)

with Y representing the vertical position and W representing the weight. The memorized target was the old target location in cases that the target had shifted location and the same as the visual target location in the unperturbed conditions. Before performing the regression, we shifted the finger position data on each trial so that the origin was always the starting position of the finger. We then normalized each movement trajectory in time and selected points in time separated by 10% of each movement's duration. This resulted in ten sets of weights for the influence of remembered and visually specified location at 10%, 20%, …, 100% of movement duration, with separate values for each subject and each high- and low-contrast condition. We computed the relative contribution of remembered location to a finger's location as the ratio of the weight given to the remembered location to the sum of weights given to the remembered and visually specified locations; that is,  

$W_{\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{y}}/(W_{\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}+W_{\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{y}}).$

(2)

Note that the relative weight given to visual location information is simply 1 minus the weight given to memorized location. Further note that the relative weight given to memory can, in principle, take on any value. When the relation between the position of the finger and remembered or visual target location is negative, the negative weight given can result in a negative relative weight given to memory or a value higher than 1. Of course, negative relations between finger position and target location do not make sense, but they can occur for noisy data. 

We used resampling to calculate a measure of the standard error on all of the weight estimates. We replicated the analysis 1,000 times by randomly sampling (with replacement) N trajectories from the set of trajectories and applying the regression analysis to these samples ( N = number of trajectories in a set). The standard deviations of the resulting weights provide estimates of the standard errors of the weight estimates. 

Eyes

The eye velocity signal was smoothed by a moving average window of 21 frames to smooth irregularities caused by random variations in the signal. Then, we detected the saccade from the trash bin toward the second target by finding the first frame in which the eye velocity exceeded 143 deg/s after the time that the finger arrived at the trash. From that frame, we searched for the frame in which the eye velocity first dropped below 57 deg/s. This point in time was defined as the end of the saccade and arrival of the eyes at the target. As we were interested in the moment after which foveal visual information to guide the hand would become available, we computed the difference between the moment that the hand started to move toward the second target and the time that the eye arrived at the target. We call this variable “peripheral hand time,” as during this time, the only visual information available to guide the moving hand is peripheral. 

We also investigated the eye movements around the time that the first target was moved toward the trash bin. This was to check whether subjects did not linger with their eyes in the target area when moving the finger toward the trash bin, in which case they might have seen the new location of the second target close to the fovea. Following the same procedure as described above, we searched for the saccade from the first target to the trash bin after the finger arrived at the first target. Then, we computed the difference between the start of this saccade and the moment that the finger left the target area with the first object. If this was less than 200 ms (the duration of the masking flicker), we would know that the gaze had left the object area at the time that the flicker ended. 

For all statistical tests, we used a level of significance of .05. 

Results

We ran a total of 10,948 trials. Out of these, 188 trials (1.7%) were rejected, mainly due to technical problems. None of the subjects reported to have noticed the target perturbations. 

Finger position

Figure 2 shows the constant and variable error, in both the vertical and horizontal directions. The hypothesis that remembered target location influences subjects' movements predicts that (a) the constant error for positive perturbations will be smaller than the constant error for negative perturbations, that is, that the finger will end up lower when the target shifted upward than when the target shifted downward, and (b) the difference between the two will be smaller for high-contrast targets than for low-contrast targets. A repeated measures ANOVA on the vertical constant error showed that there were effects of contrast—high or low: F(1, 9) = 5.60, p = .04, perturbation condition—no perturbation, upward shift, or downward shift: F(2, 18) = 51.49, p < .01, and an interaction between them— F(2, 18) = 15.67, p < .01. As can be seen in Figure 2A, the effect of perturbation condition is as predicted by the memory hypothesis: When the target shifted downward, subjects ended up high relative to where they hit unperturbed targets; if the target shifted upward, subjects end up relatively low. Thus, relative to unperturbed targets, subjects hit perturbed targets in the direction of where they used to be. The interaction with contrast suggests that this effect is stronger for the low-contrast conditions than for the high-contrast conditions. The latter was confirmed by a paired t test on the high- and low-contrast differences between the constant errors for the upward and downward perturbed targets, t(9) = 4.09, p < .01. 

Figure 2

 

Mean constant (A, C) and variable (B, D) errors in vertical (A, B) and horizontal (C, D) direction for each condition. Error bars are between-subject standard errors of the mean.

Figure 3A depicts the relative weight computed for the memorized target location over time. Figure 3B depicts the standard errors for the relative weights computed using the resampling method. Figure 3A shows that during the first part of the movements, the relative contribution of memory to weight given to memory is higher than at the end. 

Figure 3

 

Weight of memorized location over time, expressed as the ratio of the weight given to the remembered location to the sum of weights given to the remembered and visually specified locations (A) and its standard error as determined with the resampling method (B). Error bars are between-subject standard errors of the mean.

Because the eyes generally did not arrive at the target before the hand started to move (see the Eye movements section), the earliest moment that foveal target information could have started to influence the subjects' movement was after the flicker (lasting for 200 ms) and after a visuomotor delay (which is estimated to be around 120 ms: Brenner & Smeets, 1997; Prablanc & Martin, 1992). The average time that subjects needed to move from the trash to the second target was 790 ms. Thus, until around 40% of the movement time, the only target location information that could have influenced the movement was peripheral and memorized information. The estimate of 40% is rather conservative, as there may also have been an additional forward masking effect of the flicker. At 40% of the movement, the weight is 0.53 for the low-contrast target and 0.33 for the high-contrast target. At the end of the movement, these weights are 0.30 and 0.13, respectively. 

Figure 4 shows the average weights for the individual subjects at 40% of the movement (reflecting the information used in planning) and at the end of the movement (reflecting the information used in planning and online control). Different subjects appear to weigh memorized location differently. Still, all weights tend to be positive and subjects tend to weigh memorized location almost always more for low- than for high-contrast targets. Paired t tests showed that the individual memory weights were larger at 40% of the movement compared with those at 100% of the movements, t(9) = 4.50, p < .01 and t(9) = 3.91, p < .01 for high and low contrast, respectively. 

Figure 4

 

Weight of memorized location for each subject and each level of contrast at 40% of the movement (A) and at 100% of the movement (B). Error bars indicate the standard errors of the weights as computed by the resampling method.

The constant horizontal error ( Figure 2B) is not affected by contrast, perturbation condition, or the interaction between them, as shown by a repeated measures ANOVA: F(1, 9) = 0.10, p = .76; F(2, 18) = 1.54, p = .24; and F(2, 18) = 0.82, p = .46, respectively. Because the target shifted in the vertical direction, it is not surprising that perturbation condition did not affect the extent to which subjects ended up right or left from the target's center. An effect of contrast was also not expected. 

For the variable error ( Figures 2C and 2D: the standard deviation of the constant error), an effect of contrast was expected. Indeed, Wilcoxon signed rank tests on the variable errors show that a lower contrast increased the spatial variability ( p = .02 and p = .048 for the vertical and horizontal variable error, respectively), indicating that the task was more difficult when low- rather than high-contrast targets were presented. Errors toward perturbed targets were more variable than toward nonperturbed targets in the vertical direction (Wilcoxon signed rank test, p = .02) but not in the horizontal direction (Wilcoxon signed rank test, p = .32). The finding that perturbation condition affected variable error in the vertical but not in the horizontal direction suggests that on a trial-by-trial basis, subjects may have varied in the extent to which they used memorized location. 

Finger timing

Figure 5 gives an overview of the average time spent by subjects in the different phases of a trial. Repeated measures ANOVAs on the different movement and dwelling times indicate that there is usually an effect of contrast, with the time intervals being shorter for the high- than for the low-contrast targets. Only for the time spent on the trash bin and the time between leaving the trash bin and arriving at the second target was there no significant effect of contrast. In general, there are no effects of perturbation condition (the exception being the time spent on the second target) and no interactions (except for the time spent on the second target and the movement time from the starting cross to the first target). 

Figure 5

 

Mean movement and dwelling times of the finger for the low- and high-contrast conditions. Zero is the starting signal. Squares represent arrival times, whereas circles represent leaving times. To the right of the symbols, we indicated the objects in the experimental environment that correspond to these moments of arrival and leaving.

Eye movements

Typically, the eye velocity signal in a single trial shows single saccades consecutively into the direction of the object area, the trash bin, the object area, and back again toward the trash bin. 

Consistent with previous studies, we find that the eyes arrive at the target before the hand does. For the second target, the difference in arrival times is, on average, 539 ms. However, the finger starts to move, on average, 98 ms before the eyes move away from the trash bin to the second target. The mean peripheral hand time, that is, the amount of time that the finger starts to move before the eyes arrive at the target, is 206 ms ( Figure 6). Thus, during the first 206 ms of the movement, the only visual information about the target that the subjects could acquire was peripheral. In only 0.6% of the trials did the finger start to move after the eye arrived at the second target. There are no effects of contrast and perturbation on the peripheral hand time, as shown by a repeated measures ANOVA: F(1, 3) = 9.03, p = .06 and F(2, 6) = 1.65, p = .27, respectively. There is an interaction between contrast and perturbation condition, F(2, 6) = 16.42, p = .04. 

Figure 6

 

Mean peripheral hand time for each condition, that is, the time between the moment that the finger leaves from the trash bin toward the second target and the arrival of the eyes at the second target. Error bars are between-subject standard errors of the mean.

The eyes left the location of the first target to go to the trash bin, on average, 69 ms after the finger leaves. Thus, the gaze generally had left the object area long before the flicker ended (which was 200 ms after the finger left), and the possibly perturbed location of the target became visible again. 

Discussion

We investigated goal-directed movements in a situation very common in daily life, namely, when both peripheral visual information about the location of a target and memorized information from prior views are available. In particular, we measured the relative contributions of memory and online visual information to the planning of movements made during a sequential movement task. Subjects were not instructed to use memory or, in any other way, told how to perform the task, other than to complete it in a certain period. We found that when subjects initiated movements to pick up a target object, they planned their movements toward a point between the position of the target as it appeared in the periphery and the position of the target as it had appeared approximately 1 s earlier—when subjects saw the target object in the proximity of another one to which they were then moving. Subjects did this, even though in our case, endpoint feedback indicated that memory was incorrect in a third of the trials (i.e., when the target was shifted) and only the visual location was correct. Over time, the endpoint error signal could have caused subjects to rely more on vision than they usually do. Thus, our findings might underestimate the contribution of memory to the task were it performed in a stable environment. 

Subjects relied more on memory for objects that had lower visual contrast, consistent with the prediction that decreasing the reliability of visual information leads to an increased reliance on other sources of information. It also may be the case, however, that for low-contrast targets, subjects processed the initial target object locations more deeply when they picked up the first object, making the remembered location information more reliable. This could be one of the reasons why subjects took a longer time in the low-contrast condition to start the first movement and to move to the first target compared to the high-contrast condition. We quantified the relative contributions of memory and visual information by fitting a linear model to predict the position of the finger at different times during the movement as a function of the visually specified target location and the location stored in memory. For high-contrast targets, finger positions early in the trajectory were predicted by a weighted average of the remembered target location and the visual target location as it was prior to movement initiation, with the weight given to remembered location being approximately 1/2 the weight given to the visual target location. This reflects the extent to which memorized location was used relative to peripheral visual information to plan the movements. The contribution of remembered location dropped to approximately 15% of the contribution of visual target location by the end of the movement, probably because of the subjects' use of visual information for online control of the movement when the target was foveated. Diminishing the contrast of the objects increased the contribution of remembered location information so that the visual and remembered target location contributed equally to movement planning. Online control reduced the contribution of remembered location to approximately 1/2 of that of visual target location. 

It has been argued that the visual system cannot retain detailed visual information across saccades and has special difficulties with absolute spatial positions (Henderson & Hollingworth, 1999, 2003; Irwin, 1991; O'Regan, 1992). We found that the brain can and does use memorized location even when visual information is available simultaneously. This suggests that the brain stores target location precisely and reliably across saccades. The notion that little information is stored comes mainly from studies in which subjects failed to detect differences between images when an intervening saccade is made or a blank is presented between the images (Irwin, 1991; McConkie & Currie, 1996; Rensink, O'Regan, & Clark, 1997). In a sense, our study shows such change blindness as well because none of the subjects ever noticed a target perturbation (as in many other studies of visual motor control using saccade-contingent perturbations: e.g., Goodale, Pélisson, & Prablanc, 1986; Prablanc & Martin, 1992; Saunders & Knill, 2003). Subjects' movements however, reflected a strong contribution of remembered target location to their initial motor plans. Not being consciously aware of the fact that a change happened did not, in this case, mean that subjects did not store and subsequently use information about target location. 

A common idea in the literature on orienting behavior is that humans rely little on memory, choosing instead to coordinate eye and hand movements to maximize the available visual information (Ballard et al., 1995; Gajewski & Henderson, 2005; O'Regan, 1992; Rensink, 2000). In several different searching and manual tasks, an abundance of eye movements is observed, and in normal circumstances, people hardly ever grasp or touch an object without having looked at it shortly before, even if its location is known (Ballard et al., 1995; Gajewski & Henderson, 2005; Johansson, Westling, Bäckström, & Flanagan, 2001). We also found that subjects always fixate the relevant objects before arriving there with their hands. This is consistent with a strong role for visual feedback in the online control of movements (Sarlegna et al., 2003; Saunders & Knill, 2003). However, eye–hand timing is clearly not adapted to acquire the best visual information to plan the hand's movement. In our task, subjects' hands started to move, on average, before the eyes arrived on the target. This was true even for the low-contrast targets, which are more difficult to reach—as indicated by the larger variable errors and the longer movement times. Our control experiment indicated that moving the finger (long) before the eyes arrived at the target was not an artifact of the flicker. Also, in other tasks, the hand typically begins moving before or very near to the time of an orienting saccade to a target object (Binsted & Elliott, 1999; Carnahan & Marteniuk, 1991, 1994), strongly arguing that only peripheral visual information about the target is available to supplement remembered information for planning the movement. 

In accordance with the relatively late arrival of the eyes on the target and, thus, a low quality of visual location information at the beginning of the hands' movements, the finger trajectories show that the planning of the movements are partly based on memorized location information. Over the course of the movements, when foveal target location information becomes available, subjects make online adjustments toward the visually specified target location. It has to be noted, though, that with this strategy, the hand does end up at a slightly “wrong” location when the target was perturbed. Apparently, this final pointing error was small enough that it did not cause subjects to use an eye–hand coordination pattern more adapted to maximizing the use of available visual information (e.g., by arriving at the target with the gaze before moving the hand or by moving the hand more slowly). These occasional errors may have been offset by the advantages of a faster trial completion. 

While there are many studies on memory-guided reaching movements in the absence of visual location information (e.g., Admiraal, Keijsers, & Gielen, 2003; Elliott & Madalena, 1987; Heath, 2005), there is little published research on the contribution of memory when visual location information is also available. Aivar et al. (2005) clearly showed that eye–head movements are guided to remembered locations when the target is not within the field of view. They also suggested that remembered information is used to direct gaze when the target is visible in the periphery. In their study, subjects made eye–head movements to a working area containing elements that subjects used to build a model. Unnoticed by the subjects, the building blocks could switch location. For some trials, subjects' gaze landed at the old memorized location of the element they wanted to fetch, even when that location “was currently visible in the retinal image.” However, it is not clear whether there actually was peripheral information available that subjects could have used in those trials because some of the blocks only differed in color, which is poorly coded in the peripheral retina. Also, in the trials where visual information might have been available, the task was more concerned with target selection rather than target location. In the context of viewing scenes, Chun and Nakayama (2000) describe two effects that show the use of memory in guiding attention and eye movements. Contextual cueing is demonstrated by an advantage of finding a target among distracters if that particular configuration of target and distracters had been presented before, even if the subject is at chance at discriminating old configurations from new ones. Priming of popout is the advantage of attending or saccading to an odd-colored target to identify its exact shape if it has the same color or location as in trials (shortly) before. Chun and Nakayama propose that implicit (unconscious) visual memory traces are responsible for a continuity of visual processing across fixations and an efficient guidance of attention and gaze. Our experiment shows that unconscious memory is used not only for gaze and attention but also for planning hand movements, even when peripheral visual information is available at the same time. 

Acknowledgments

We would like to thank Brian McCann for his contribution to the experimental setup, Ted Twietmeyer for constructing the EOG equipment, and Beth DeGrush and Jacquelyn Spector for their contribution to data acquisition. 

Commercial relationships: none. 

Corresponding author: Anne-Marie Brouwer. 

Email: [e-mail]. 

Address: 275 Meliora Hall, Rochester, NY 14627, USA.