Decoupling eye and hand movement control: Visual short-term memory influences reach planning more than saccade planning

Abstract

When reaching for objects, humans make saccades to fixate the object at or near the time the hand begins to move. In order to address whether the CNS relies on a common representation of target positions to plan both saccades and hand movements, we quantified the contributions of visual short-term memory (VSTM) to hand and eye movements executed during the same coordinated actions. Subjects performed a sequential movement task in which they picked up one of two objects on the right side of a virtual display (the “weapon”), moved it to the left side of the display (to a “reloading station”) and then moved it back to the right side to hit the other object (the target). On some trials, the target was perturbed by 1° of visual angle while subjects moved the weapon to the reloading station. Although subjects did not notice the change, the original position of the target, encoded in VSTM, influenced the motor plans for both the hand and the eye back to the target. Memory influenced motor plans for distant targets more than for near targets, indicating that sensorimotor planning is sensitive to the reliability of available information; however, memory had a larger influence on hand movements than on eye movements. This suggests that spatial planning for coordinated saccades and hand movements are dissociated at the level of processing at which online visual information is integrated with information in short-term memory.

Introduction

When pointing, grasping, or reaching to touch objects, humans typically make saccadic eye movements to fixate the target object just prior to or early in the hand movement (Abrams, Meyer et al., 1990; Biguer, Jeannerod et al., 1982; Prablanc, Echallier et al., 1979; Prablanc, Pelisson et al., 1986). While there is some variability in the relative timing of eye and hand movements during natural movements (Abrams et al., 1990; Carnahan & Marteniuk, 1991; Pelz, Hayhoe et al., 2001), depending, for example, on specific task demands, the eye typically fixates target objects just before or slightly after the beginning of hand movements but well before their completion (Frens & Erkelens, 1991; Helsen, Elliott et al., 2000; Starkes, Helsen et al., 2002) and then maintains fixation on the target of a hand movement until the movement has been completed, even in a sequential movement task in which subjects must touch a sequence of targets (Neggers & Bekkering, 2000, 2001, 2002). 

The picture that emerges of eye–hand coordination is that the CNS usually ensures that the targets of hand movements are fixated during the entire last half (at least) of the movements so that reliable visual information is available for online control (Elliott, 1992). Because of the relative timing of saccade and hand movement initiation, the CNS has essentially the same information about target objects available to plan both eye and hand movements—that is, peripheral visual information and information stored in visual short-term memory (VSTM) from previous fixations, either to the target itself or to other objects involved in ongoing behavior. This suggests that the CNS may use a common spatial representation of targets to plan both eye and hand movements. Evidence for the hypothesis that a common spatial representation guides saccade and hand movement planning has been equivocal. Some studies find little correlation between eye and finger endpoints when endpoint variance is a result of simple variable error, even when the target was extinguished on movement onset (Biguer, Prablanc et al., 1984; Prablanc et al., 1979). Other studies have shown stronger correlations when endpoint variability is created by changes in illusory configurations (e.g., the Muller–Lyer illusion; Binsted, Chua et al., 2001; Binsted & Elliott, 1999; de Grave, Franz et al., 2006; Mack, Heuer et al., 1985). 

Here, we take a new approach to testing the common spatial command hypothesis for coordinated eye and hand movements. It is based on the finding that the CNS integrates position information from visual short-term memory (VSTM) with the immediately available peripheral visual information about a target when generating initial hand movement plans (Brouwer & Knill, 2007, 2009). The common command hypothesis predicts that information in VSTM will influence saccade plans as much as hand movement plans during the same coordinated movements. In the experiment described here, subjects naturally executed temporally coordinated saccades and hand movements in a sequential pointing task; however, their saccades were significantly less influenced by information in VSTM than were the initial kinematics of their hand movements. This suggests a decoupling of the computations driving saccade and hand movement planning, whereby separate spatial estimations contribute to the two motor plans, at least at the stage where online visual information is integrated with information from VSTM. 

Methods and materials

Overview of experimental logic

Subjects viewed a virtual tabletop containing a virtual magnetic weapon and a target (on the right-hand side of the table) and a reloading station (on the left-hand side of the table; see Figure 1). Subjects were instructed to start each trial with their finger positioned on a cross on the bottom of the table. When this cross spun, signaling the start of the trial, subjects were to successively pick up the virtual weapon (by touching their finger to the weapon), move it to the reloading station, then move the loaded weapon back to touch the target, which would explode if and when contact was made. Unknown to the subjects, in a third of the trials, the target object shifted 1 cm (∼1 deg of visual angle) up or down while the subject was transporting the weapon to the reloading station. This created a situation in which information about the position of the target stored in visual short-term memory conflicted with its position in the peripheral visual field when a subject's finger and eyes were positioned over the reloading station. A brief flicker of the display at the onset of the first movement to the reloading station masked the perturbations so that subjects were unaware that they ever occurred. Similarly, a brief flicker at the beginning of the second movement to the target precluded visual online adjustments of the hand movement during its early phase. In order to estimate the weight that subjects gave to remembered and visually specified target location to plan saccades to the target for the second movement, we regressed the position of the eye gaze at the end of the initial orienting saccade against the initial (remembered) position of the target and its final (visual) position. Similarly, we regressed subjects' finger positions at different points in time during the second movement against the initial position of the target and its final position. The regression weights to remembered position derived from saccade endpoints provided measures of the influence of remembered position on saccade planning, while the regression weights to remembered position derived from finger positions early in the movement (before online control could take effect) provided measures of the influence of remembered position on hand movement planning (see Equation 4 in the Data analysis section for details.) 

Figure 1

 

(A) The viewing arrangement. Subjects viewed the stimulus through a mirror, so that the entire tableau of start position, weapon, target, and reloading station appeared to be resting on the table positioned below the mirror. Optotrak markers mounted on a thimble worn by the subject were used to track the position of the fingertip. (B) A schematic sequence of four frames from a trial in the experiment: the start when the subject's finger is touching the start cross, when the subject touches and picks up the weapon, when they touch the reloading station, and when they touch and explode the target. Note that on the trial depicted here, the target shifted down by 1 cm after the subject picked up the weapon. This is indicated by the arrow superimposed on the stimulus in the third frame; the arrow did not appear in the actual stimulus.

Participants

Eleven right-handed adults, six males and five females, aged 19–31, were recruited from the University of Rochester participant pool. Subjects were naive to the purpose of the study, gave informed consent, and received $10 per hour of participation. All subjects had self-reported normal or corrected-to-normal visual acuity. 

Equipment and setup

Participants were seated in front of a mirror and a frontoparallel table as diagrammed in Figure 1A. The mirror occluded the subject's view of their hand and the table, and it also reflected the virtual world that was projected by a CRT monitor above. Participants' heads were stabilized by a forehead rest and a bite bar containing a dental impression and viewed the display monocularly by patching the right eye. The starting cross, weapon, target, and reloading station were made to appear on the table by rendering them at the appropriate locations and orientations. A virtual finger was rendered as a cylinder with a rounded tip at the position and orientation of the subject's real finger. Subjects viewed the display from an optical distance of approximately 52 cm (with small variations from subject to subject). The physical table was positioned to be co-aligned with the monitor's optical image. 

Participants wore a metal sleeve on the right index finger, which was fitted with three Optotrak markers mounted on a small flag raised above the finger sleeve. Marker positions were tracked at 120 Hz by an Optotrak 3020 camera system (NDI, Waterloo, Ontario, Canada). The information was used online to compute the position and pose of the finger. We used linear extrapolation of the finger's position to correct for the approximately 20-ms delay between recording the finger's position and the appearance of the virtual finger in the display; thus, the virtual finger moved in real time with the actual, unseen finger. When visually compared with one's real finger by viewing the display through a half-silvered mirror, the only notable errors in the virtual finger appeared at the very beginnings and ends of movements, when accelerations and decelerations were high. The system was calibrated by having subjects touch 36 points with known positions on the table at the beginning of each daily session. 

Two steel plates were positioned left to right on the table. The weapon and target were projected over the right plate and the starting cross and the reloading center were projected over the left plate. The plates were connected to a 5-V source and to a Northern Digital Optotrak Data Acquisition Unit II that recorded the voltage across each plate at 120 Hz. The steel sleeve worn over the index finger acted as a ground, so that by measuring the voltage of the plates we acquired precise measurements of the time that the finger left and arrived at the left or the right plate; hence, the timing of the beginnings and ends of the sub-movements involved in the task. 

Eye gaze position was recorded by an Eyelink infrared camera using pupil location and corneal reflection at 250 Hz. A nine-point eye calibration preceded each block of 60 trials and a single-point drift correction was enacted every 10 trials and as needed (when the experimenter noticed significant drift between the fixations and the objects being fixated). 

Stimuli

Two stimulus conditions were used in the experiment. In the “near” condition, distances between both the weapon and the target, on the one hand, and the reloading station, on the other hand, were ∼13.5 cm (∼14.5 deg of visual angle). In the “far” condition, distances were ∼27.6 cm (∼28 deg of visual angle). The starting cross was always located at the bottom of the virtual tabletop, 11 cm (12 deg of visual angle) below the horizontal midline and 2 cm (2.2 deg of visual angle) to the right of the vertical midline. Slight variations in the relative positions of other objects in the display were rendered from trial to trial in order to discourage rehearsal of a consistent motor command. Figure 2 schematically shows the possible locations of the weapon, target, and reloading station in the “far” condition. To create the arrangement of objects in the scene on any given trial, we first positioned the reloading station along the horizontal midline of the display at the left side of an imaginary circle centered at the middle of the display with a diameter of 28 cm (∼29 deg of visual angle). The target and weapon were located along the right edge of this circle. The weapon was randomly positioned at an angle around the perimeter of the circle randomly chosen from a uniform distribution between 5 and 25 degrees of theta either above or below the horizontal midline. The target was randomly positioned at an angle along the perimeter of the circle between 5 and 25 degrees of theta from the horizontal midline on the opposite side of the horizontal midline from the weapon. The entire configuration of weapon, target, and reloading station was randomly rotated around the middle of the imaginary circle by an angle drawn uniformly from the range ±2.5 degrees of theta. With this layout, the distance between the weapon and the target ranged from 10 to 50 degrees of theta around the perimeter of the imaginary circle. This was equivalent to distances between 2.44 and 11.83 cm on the tabletop (2.70–12.82 deg of visual angle). Distances between either the weapon or the target and the reloading station varied between 27.34 and 27.95 cm (27.74 and 28.26 deg of visual angle). 

Figure 2

 

A schematic showing how positions for the weapon, target, and reloading station were randomly selected on each trial. In the “far” condition, the weapon was positioned randomly in one of the arcs shown on the right-hand side of the figure, as described in the text. The target was randomly positioned within the other arc. Here, the target is shown on the top and the weapon on the bottom. The reloading station was then placed at the left of the figure along the horizontal midline of the display. The entire configuration was rotated by an angle randomly and uniformly drawn from the range ±2.5 degrees around the center of the display. In the near condition, this spatial configuration, including the ranges within which the target and weapon were positioned, was scaled down by 1/2.

In the “near” condition, the imaginary circle had a diameter of 14 cm (∼15 deg visual angle) so that both the weapon and the target were positioned 14 cm away from the reloading station. In order to maintain a similar range of absolute distances between the weapon and target as in the “far” condition, they were randomly positioned between 20 and 40 degrees above or below the horizontal midline. Distances between the weapon and the target ranged from 4.79 to 9.0 cm on the tabletop (5.26–9.82 deg visual angle). Distances between either the weapon or the target and the reloading station varied between 13.16 and 13.79 cm (14.20 and 14.85 deg visual angle). As in the far condition, the entire configuration of objects was rotated by a random angle between ±2.5 degrees around the screen center. 

The weapon was a 7-mm-wide square, the target was a 7-mm-diameter circle (∼0.8 deg of visual angle), and the reloading station was a plus sign with a diameter of 12 mm (∼1.3 deg of visual angle). All visual objects were displayed in red (12.8 cd/m² as measured with a Minolta luminance meter LS-110) on a gray background (5.35 cd/m²). By using only the fast red phosphor, we minimized binocular cross talk between left and right images appearing through stereo glasses in other versions of this experiment. Figure 3 shows an example of the entire stimulus tableau at key points in a sample trial, exactly as they appeared to subjects. 

Figure 3

 

An example of the stimulus display actually seen by a subject at four different points in time during a trial. Left to right, then top to bottom: Just before picking up the weapon, successful pickup of the weapon, successful touch onto the reloading station, and successful touch on the target.

Procedure

The experiments were performed in a darkened room. A subject started each trial by positioning the right index finger on the starting cross. After 500 ms, the cross spun, signaling the subject to pick up the weapon. If the movement started before 100 ms after the start signal, a “too early” warning appeared and the trial was repeated later. On touching the weapon, it “magnetically” stuck to the finger, as indicated by the target briefly increasing in size. The criterion for touching the target was that the fingertip had to be within 10 mm of the center of the target. Subjects were instructed to touch the weapon to the reloading station and then to place the weapon on the target. At the moment that the subject's finger left the table after picking up the weapon, we presented a 167-ms masking flicker, consisting of a black and gray checkerboard pattern counterphase flickering at 15 Hz. Luminance values for the checkerboard pattern were 1.4 and 10.4 cd/m². On two thirds of trials, nothing changed during the flicker, but on one-third of trials, the target shifted vertically 1 cm up or down on the screen. When the subject touched the weapon to the reloading station (to within 2.5 mm of the middle of the reloading station), the cross at the reloading station spun around indicating that the weapon was loaded and the subject could move the weapon to hit the target. Finally, the subject moved the weapon from the reloading station to the target. When the subject lifted his or her finger from the table, another 160-ms flicker was presented. The aim of this flicker was to make sure that the first portion of the movement reflected information used in the planning phase only and could not be adjusted on the basis of newly acquired foveal target location information. When the subject hit the target (to within 2.5 mm of the center), they were rewarded with a visual explosion of the weapon and target. This ended the trial, at which point the subject moved back to the start cross to begin the next trial. Subjects were asked to move quickly but precisely. If the complete trial was not finished within 6 s or an individual sub-movement lasted more than 2 s, a message appeared that this was too slow and the trial was repeated later. (Typical sub-movements were less than 1 s each. See Supplementary materials for details.) 

All subjects started the experiment each day with a practice block of 30 trials, which contained 15 unperturbed trials for each of the near and far conditions. These data were not analyzed. Each experimental block consisted of 18 trials each in the near and far conditions, randomly interleaved. On 3 of the 18 trials, the target was perturbed up during the movement to the reloading station, and on 3 trials, it was moved down; thus, a block contained a total of 24 unperturbed trials and 12 perturbation trials. Subjects performed five experimental blocks in each of 5 days of testing. The first day's data were discarded as training data, leaving 720 total trials over the final 4 days of testing for analysis. 

After the complete experiment, subjects were asked whether they had noticed the target shifting up or down on any trials. None of the subjects reported that they had, and they were surprised to learn these perturbations actually occurred 33% of the time. 

Data analysis

Eye movements

Eye gaze position signals were smoothed with a Gaussian filter with a standard deviation of 8 ms before analysis. Occasionally, the eye tracker lost track of the eye due to a blink or other unknown factor. This was indicated in the gaze position signal by overflow values. When the track loss occurred during the phase of the trial critical for data analysis (the saccade from the reloading station to the target), we discarded the trial from further analysis (see Trial selection section below). The beginning of a saccade was marked wherever the smoothed eye velocity exceeded 25 cm/s in tabletop coordinates (approximately 27.5 deg/s). The end of the saccade was marked as the next point at which the eye velocity dipped below 100 cm/s (approximately 110 deg/s). We used a conservative estimate for the end of a saccade in order to ensure that none of the gaze positions used to estimate the influence of visual short-term memory on saccade planning could have been influenced by visual information available at the end of the saccade. A small number of anomalous saccades had long, low-velocity tails that led to excessively high estimates of saccade end-times for the saccades. 

Trial selection

We selected a subset of trials for analysis based on gaze position and eye movement criteria designed to (a) eliminate trials with “bad” eye movement data, either because of losing the eye tracking signal or because of data corruption by blinks during or near to the time of saccades, and (b) ensure that the information available to the subject for planning their hand movement was the same as was available for planning their eye movement. We used the following criteria to select trials containing measured orienting saccades from the reloading station to the target: 

1.  
   A valid saccade was measured after the finger successfully placed the weapon at the reloading station. Besides matching the velocity criteria for a saccade noted above, the maximum eye velocity as recorded by the tracker had to be less than 1000 cm/s (∼1,102 deg/s), thus eliminating blinks.
2.  
   The gaze position recorded by the eye tracker was within 4 cm (4.4 deg) of the reloading station 40 ms before the saccade start time (the 40-ms shift corrected for the fact that the threshold criterion used to mark the beginning of a saccade created a positive bias in the measured start time).
3.  
   The length of the saccade to the target was between 50 and 150% of the distance between the reloading station and the target.
4.  
   The eye tracker did not lose track of the eye during the saccade for more than 5 frames (20 ms) during the saccade.
5.  
   To ensure stable fixation for 40 ms after the estimated end of the saccade, valid trials must have a standard deviation of recorded gaze positions for 10 frames (40 ms) after the marked saccade endpoint of less than 1 cm (1 deg).

To ensure that we only analyzed trials in which subjects did not have foveal information about the target available for planning the finger movement, we enforced two further constraints: 

1.  
   The eye was never within 10 cm of the circular target at any point between the end of the first flicker (during the hand movement to the reloading station) and the time subjects were given the signal that the weapon had reloaded. This eliminated possible look-ahead saccades to the perturbed target.
2.  
   The finger must leave the reloading station (broke contact with the contact plate) to move to the target before the end of the saccade to the target.

Linear regression

In order to estimate the influence of VSTM on saccade movement planning, we modeled the gaze position in tabletop coordinates at the end of the saccade as a linear function of the target position at the time of weapon pickup (prior to any possible perturbation—the “remembered position”), the target position at the time subjects touched the reloading station (after any possible perturbation—the “visual position”), and the gaze position just prior to the saccade. The latter was estimated by averaging the recorded gaze position over 10 eye tracker frames (40 ms) 20 ms prior to the marked beginning of the saccade. The 20-ms offset was used to ensure that only gaze positions during the pre-saccade fixation period were incorporated into the average. Perturbations were entirely in the y direction, so we focused our analysis on the vertical position of gaze (in tabletop coordinates). 

The model for vertical gaze position takes the following form: 

$Y_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{e}\mathrm{y}\mathrm{e}\mathrm{)}}=w_{\mathrm{m}\mathrm{e}\mathrm{m}}{Y}_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{)}}+w_{\mathrm{v}\mathrm{i}\mathrm{s}}{Y}_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{(\mathrm{v}\mathrm{i}\mathrm{s})}+\beta Y_{\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}}^{\mathrm{(}\mathrm{e}\mathrm{y}\mathrm{e}\mathrm{)}}+\gamma +\mathrm{N}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},$

(1)

where Y _target ^(eye) is the Y-coordinate of gaze in tabletop coordinates at the marked endpoint of the saccade, Y _target ^(mem) is the Y-coordinate of the target prior to the perturbation (the remembered position), Y _target ^(vis) is the Y-coordinate of the target after the perturbation (the visually specified position at the time of saccade planning), Y _reload ^(eye) is the position of the gaze just prior to the saccade to the target (as the location of the reload target differed from trial to trial, as shown in Figure 2), and γ is a constant bias term. The noise term captures the aggregated affects of perceptual noise, motor noise, and eye tracker errors on recorded gaze positions at the end of a saccade. For non-perturbation trials, Y _target ^(mem) = Y _target ^(vis) and Equation 1 reduces to 

$Y_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{e}\mathrm{y}\mathrm{e}\mathrm{)}}=\alpha Y_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}+\beta Y_{\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}}^{\mathrm{(}\mathrm{e}\mathrm{y}\mathrm{e}\mathrm{)}}+\gamma +\mathrm{N}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},$

(2)

where Y _target is simply the position of the target and α = w _mem + w _vis. To isolate the effect of sensorimotor biases and the relative weighting of memory and vision, we can rewrite Equation 1 as 

$Y_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{e}\mathrm{y}\mathrm{e}\mathrm{)}}=\alpha [w_{\mathrm{m}\mathrm{e}\mathrm{m}}^{*}{Y}_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{)}}+(1-w_{\mathrm{m}\mathrm{e}\mathrm{m}}^{*}\left)Y_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{)}}\right]+\beta Y_{\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}}^{\mathrm{(}\mathrm{e}\mathrm{y}\mathrm{e}\mathrm{)}}+\gamma +\mathrm{N}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},$

(3)

where w _mem* reflects the normalized (proportional) contribution of visual short-term memory to the gaze position at the end of the saccade. In order to estimate w _mem*, we regressed the recorded gaze positions of the eye at the endpoint of saccades to the target against the remembered and current positions of the target using Equation 1 and normalized the weights using 

$w_{\mathrm{m}\mathrm{e}\mathrm{m}}^{*}=\frac{w_{\mathrm{m}\mathrm{e}\mathrm{m}}}{w_{\mathrm{m}\mathrm{e}\mathrm{m}}+w_{\mathrm{v}\mathrm{i}\mathrm{s}}}.$

(4)

 

We used a similar model to analyze the relative influence of remembered and visual target position information on subjects' hand movements. Rather than simply fitting the model to finger position data at the end of the movement (after online corrections have brought the finger into alignment with the visually specified target position), we applied the model to the finger position at regularly spaced time intervals during the movement. To do this, we first interpolated the finger movement data from the time of liftoff from the contact plate at the reloading station to the time the finger first touched the contact plate near the target, thus normalizing movement trajectories to a common duration of 0–100%. We used a model equivalent to Equations 1 and 4, but applied to finger position data, to fit normalized weights to remembered position as a function of normalized time throughout the movement. The resulting regression model was given by 

$Y^{\mathrm{(}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{r}\mathrm{)}}\left(t\right)=w_{\mathrm{m}\mathrm{e}\mathrm{m}}{Y}_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{)}}\left(t\right)+w_{\mathrm{v}\mathrm{i}\mathrm{s}}{Y}_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}^{\mathrm{(}\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{)}}\left(t\right)+\beta Y_{\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}}^{\mathrm{(}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{r}\mathrm{)}}+\gamma +\mathrm{N}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e},$

(5)

where Y ^(finger)(t) is the Y-position of the fingertip in tabletop coordinates at normalized time t (0 < t ≤ 100); thus, Y ^(finger)(50), for example, is the position of the fingertip halfway through the movement. The weights are functions of normalized time, and the relative contribution of remembered target position to fingertip position at time t is given by 

$w_{\mathrm{m}\mathrm{e}\mathrm{m}}^{*}\left(t\right)=\frac{w_{\mathrm{m}\mathrm{e}\mathrm{m}}\left(t\right)}{w_{\mathrm{m}\mathrm{e}\mathrm{m}}\left(t\right)+w_{\mathrm{v}\mathrm{i}\mathrm{s}}\left(t\right)}.$

(6)

 

Results

It was discovered while running one of the subjects that he was severely amblyopic in the eye we were using to display stimuli and record eye movements. In another subject, the experimenter noticed anomalous behavior in the eye tracker, with the displayed position of the eye jittering by an abnormally large amount. Both of these subjects were removed from the analysis, though all results reported here were the same when these subjects' data were included. 

Summary movement statistics

Figure 4 shows sample gaze position trajectories for both near and far conditions for one subject in unperturbed trials. For purposes of these figures, gaze positions were projected onto the axis between the reloading station and the target on each trial. The trajectories are lined up at the time marked as the saccade onset. The solid lines indicate the periods of the eye movements marked as saccades. The dotted lines are the gaze positions and velocities preceding and following the saccades. Plots for all of the subjects appear qualitatively like the ones shown here. These data show that our analysis code is accurately finding the beginning and ends of saccades. They also show that the high threshold used to mark the ends of saccades, as expected, underestimates the timing of saccade endpoints. Since our purpose is to use saccade endpoints to estimate the contribution of visual short-term memory to saccade planning, the conservative selection of saccade endpoints ensured that none of the eye positions used for the estimates corresponded to times after visual information became available to make corrective saccades (see Supplementary materials for further timing data and movement kinematics). 

Figure 4

 

Plots of eye movement kinematics for saccades from the reloading station to the target. (A, B) Eye position and velocity trajectories for the “near” condition. (C, D) Eye position and velocity trajectories for the “far” condition. Black lines indicate the time period between the marked beginnings and ends of saccades.

As noted in the Data analysis section, we discarded trials from further analysis based on a set of eye movement criteria. We discarded an average of 48.3 trials (out of 360) per subject in the near condition and 95.3 trials (out of 360) in the far condition. By far, the largest proportion of discarded trials resulted from not detecting a valid saccade to the target from the reloading station after the reloading station had been successfully touched. This was typically due to a loss of the eye track or an eye blink during the orienting saccade. This accounted for 81.2% of the discarded trials in the “near” condition and 87.3% of the discarded trials in the “far” condition. 

Regression analysis

Figure 5 shows the relative weights to the remembered target position derived by fitting the regression model expressed in Equations 5 and 6 to the finger movement data, plotted as a function of the normalized time during the movement. The weight functions are relatively flat early in the movement but decrease to a small positive value at the end. This pattern mimics that found in our previous work using a similar paradigm to study the influence of visual short-term memory on hand movements (Brouwer & Knill, 2007, 2009). Since online visual information was not available until after the flicker, the initial portions of these curves reflect the influence of information in visual short-term memory on motor planning. We would like to use the regression weights as early in the movement as possible as a measure of the contribution of VSTM to movement planning. As shown in Figure 5, however, the standard errors on weight estimates are very large early in the movement and shrink as a function of time into the movement—a natural consequence of the fact that hand paths to different estimated target locations diverge more and more as a function of time into the movement. We would therefore like to use weights computed as late in the movement as possible to gain statistical power. Since it takes at least 100 ms for corrections to target perturbations to appear in movement kinematics (Prablanc & Martin, 1992), we used the regression weights at a point in the time-normalized trajectories that corresponded on average to a delay of 97 ms after the first post-flicker video frame in the far condition (34% of the way through the hand movement) as an indicator of the influence of VSTM on motor plans. In order to compare near and far conditions, we used weights computed at the same point in the time-normalized trajectories for the near condition. This corresponded, on average, to a 49-ms delay from the end of the flicker. 

Figure 5

 

The average relative influence of the position of the target prior to a perturbation as measured by the weights derived from the regression (Equations 5 and 6) applied to data from the “far” condition (upper curve) and the “near” condition (lower curve). We first normalized each movement so that finger positions were represented as a function of the proportion of total movement time. We then fit the regression model to the finger positions at 1%, 2%, …, 100% of the movement. The gray regions represent ±1 standard error of the mean (across subjects). Weights for times prior to 20% of the way through a movement are not shown because the standard errors are too large to make them meaningful.

Eye position data early in saccadic movements to the target were too variable to derive reliable estimates of the relative influence of visual short-term memory on saccade planning. Instead, we used the position of the eye at the “marked” end of the saccade to estimate. Because of the speed at which they are executed, saccadic eye movements are ballistic and are not subject to an online control feedback loop (Westheimer, 1954), unlike hand movements. Thus, it was appropriate to use the endpoint of saccadic movements to estimate the influence of visual short-term memory on saccade planning. As described above, we used a conservative speed threshold to mark saccade endpoints (generally just prior to the true saccade endpoints—see Figure 4) to avoid any possibility of the data being corrupted by small corrective eye movements. Figure 6A shows average weights across subjects for the hand and eye as a function of “near” vs. “far” condition. A two-way, repeated measures ANOVA revealed significant main effects of stimulus condition (near vs. far) (F(1, 1) = 13, p = 0.0069) and of effector (eye vs. hand) (F(1, 1) = 5.5, p = 0.047), with no significant interaction (F(1, 1) = 0.67, p = 0.437). Individual t-tests showed that the weights were significantly different from 0 for all but the eye in the “near” condition (eye–“near,” T(8) = −0.916, p = 0.386; eye–“far,” T(8) = 2.84, p = 0.022; hand–“near,” T(8) = 2.70, p = 0.027; hand–“far,” T(8) = 2.79, p = 0.023. (See Supplementary materials for a subject-by-subject comparison of eye and hand weights in the far and near conditions.) 

Figure 6

 

(A) The relative influence of remembered position averaged across subjects, for saccades and hand position 34% through the movement by time. (B) The same as (A) but with trials containing early fixations on the target removed from the analysis.

A qualitative analysis of eye movement data shows two qualitatively different eye movement behaviors early in task completion, prior to picking up the weapon. In most trials, subjects saccade directly to the weapon, but in a minority of trials, they make an initial saccade to the target before finally fixating the weapon prior to pickup. This raises the possibility that early fixation of the target may improve memory encoding and thus interact with how subjects use information in VSTM to plan final movements from the reloading station to the target; in particular, it raises the possibility that memory only contributes to planning for one or the other motor system when the target was fixated during initial encoding of its position. To test this possibility, we marked trials in which subjects fixated the target prior to picking up the weapon and moving it back to the reloading station (at which point subjects always made saccades to the reloading station), removed them from the data set, and redid the regression analysis. 

Trials with early target fixations were defined as ones in which a subject's eye position remained within 3 cm of the target at any point prior to weapon pickup for a duration of 80 ms or more. On average, 23.1% of trials in the near condition and 26.7% of trials in the far condition contained early target fixations, with no subject having more than 41.1% of trials with early fixations in any condition. Figure 6B plots the average memory weights for eye and finger movements computed after removing these trials. Removing trials with early target fixations does not change the results and repeating the statistical test described above on the reduced data set gives equivalent results (see Table 1). Unfortunately, the small number of trials containing early fixations precludes any meaningful statistical comparison of behavior between trials containing early target fixations and trials not containing them. 

Table 1

 

The estimated weight given to memory for planning hand movements, calculated at different time points or position points, or with/without including trials with look-ahead fixations, and the estimated weight given to memory for planning eye movements.

Movement type	Hand				Saccades
Condition	34% of time	34% of time no look-aheads	50% of time	56% of distance	Endpoint
Near	0.19 ± 0.07	0.17 ± 0.08	0.18 ± 0.05	0.26 ± 0.04	−0.023 ± 0.03
Far	0.34 ± 0.12	0.45 ± 0.13	0.34 ± 0.07	0.41 ± 0.12	0.24 ± 0.08

 

Values after ± indicate standard errors on weight estimates.

Our analysis of the influence of VSTM on finger movement planning suffers from two potential problems. First, individual subjects and individual trials had trajectories of different durations, so the absolute time points going into the weight computations for the 34% point in normalized time varied (between-subjects standard deviation = 43.7 ms, average within-subjects standard deviation = 39.6 ms). Half of the trajectory data contributing to the weight estimates were taken from absolute time points greater than 97 ms after the end of the flicker and fully 16% of the total number of data points in the regression were taken from times greater than 150 ms after flicker. It should be noted that the 100-ms reaction time estimate for online corrections is taken from experiments using large perturbations, unlike the small 1-cm perturbations used here. In experiments using such small perturbations, detectable effects of online corrections are seen approximately 150 ms after the perturbation (Saunders & Knill, 2004). This still leaves 16% of the data points taken from times greater than this upper bound on the delay at which effects of online corrections would appear in subject trajectories (from trials with particularly long duration trajectories). 

The influence of any such corrections on the regression, however, would be to shrink the influence of VSTM, since they would be based on visual information about the post-perturbation target position; thus, the regression weights reflect lower bounds on the influence of VSTM on hand movement planning. Any effects of online correction on these regression weights would shrink the observed size of the difference between hand and eye movement planning. Furthermore, if we use a time point in the normalized trajectories at which the standard errors in weight estimates has decreased significantly, but before the apparent effects of online corrections reveal themselves (50%—see Figure 5), the statistical effects remain the same but with much improved significance levels (see Tables 1 and 2). 

Table 2

 

The ANOVA results for condition × effector (near/far vs. eye/hand), calculated for different time and position points along the hand movement trajectory.

Factor	34% of time	50% of time	56% of distance
*Near vs. far	*F = 13 (p = 0.0069)	*F = 23.0 (p = 0.0014)	*F = 9.75 (p = 0.014)
*Eye vs. hand	*F = 5.5 (p = 0.047)	*F = 23.0 (p = 0.0014)	*F = 16 (p = 0.0040)
Interaction	F = 0.67 (p = 0.437)	F = 1 (p = 0.347)	F = 1.5 (p = 0.256)

 

Asterisks indicate significant terms and show that our pattern of results is independent for several methods of isolating the hand movement planning.

The second issue associated with the measurement of planning weights relates to the proper way to normalize the trajectories. We normalized the trajectories in time, effectively assuming time-scale invariance of whole hand movements. The existence of a late-epoch, low-velocity phase in subjects’ movements, during which the majority of online corrections occur, suggests that while the early phase of movements may be approximately time-scale invariant, the entire trajectories are not. An alternative would be to assume that the y-component of hand trajectories was invariant to the distance traversed either along the path or in the horizontal direction. This would mitigate the problem posed by time-varying late-epoch corrections, since they occur largely in the immediate neighborhood of the target. We therefore reran the regression using data points from corresponding points in subjects' trajectories either in distance along the path to the target or in distance in the horizontal direction. We chose a point 56% of the way along the trajectory for the analysis, since it corresponded to an average delay of 100 ms following the flicker in the far condition (35 ms after the flicker in the near condition). The results of the analysis, including ANOVA results, were essentially the same as derived from the time-normalized data (see Tables 1 and 2). 

Discussion

Influence of VSTM on movement planning

Subjects' hand movements to the target clearly show an impact of the remembered position of the target even when peripheral visual information was available for planning and executing the movements (see Figure 6). As in previous studies using the current task paradigm, nothing about the task or instructions explicitly biased subjects to use VSTM, yet they do so, even in trials in which they do not fixate the target prior to reaching to hit it with the weapon. The influence of a target's remembered spatial location on hand movement depended on the reliability of the online visual information available, increasing with eccentricity, mimicking similar effects of target contrast found in previous studies (Brouwer & Knill, 2007, 2009). This suggests that using VSTM in this task was an automatic behavior in response to simple accuracy demands. We also found a significantly stronger weight given to memory for hand movements than for eye movements in both conditions, with a stronger weight to memory in the far condition. This indicates that differences in the relative contribution of visual short-term memory to different sensorimotor behaviors are likely to reflect differences in the reliability of stored information imposed by stimulus structure and task constraints (e.g., the time between when information is stored in memory and when it is used). 

Information in visual short-term memory also influenced orienting saccades in the far condition—subjects' saccades to the target were biased toward the remembered target location. As with hand movements, the influence of VSTM on saccade planning was significantly less in the near condition than the far condition. The measured memory weight in the near condition was extremely low and was not significantly different from 0. While we cannot accept the Null hypothesis that the mean weight is 0, the upper 95% confidence bound was only 0.0436, meaning that any real effect of memory on saccade planning for the near targets was minimal. Most notably, VSTM had a significantly weaker influence on the initial saccade to a target than it did on the initial hand movement plan. The results imply that at least in some conditions, the CNS uses information in VSTM to help plan saccades even when peripheral visual information about target location is available; however, remembered location information has significantly less influence on saccade planning than on hand movement planning when coordinating both types of movements to a target. 

Implications for the “common command” hypothesis of eye–hand coordination

Early observations that eye and hand movements to a target during simple pointing tasks were initiated at close to the same time (Biguer et al., 1982) led to the proposal that both movements are driven by a common command (Bizzi, Kalil et al., 1971; Herman & Maulucci, 1981). Most research on this proposal has focused on the relative timing of eye and hand movements. Early work using simple speeded pointing tasks suggested a relatively fixed coordination strategy, with saccades initiated just before hand movements (Prablanc et al., 1979) and fixation on targets enforced throughout the duration of hand movements (Frens & Erkelens, 1991; Neggers & Bekkering, 2000, 2001, 2002); however, more recent work has shown that the CNS uses a more flexible and adaptive approach to coordinating eye and hand movements tied to the informational demands of the current task (Land, Mennie et al., 1999; Pelz et al., 2001; Sims, Jacobs et al., 2011). 

The fact that hand movements are typically initiated before the end of the orienting saccade to a target (Helsen et al., 2000) means that the information available to plan both eye and hand movements to a target is the same peripheral visual information and information stored in VSTM. This suggests that the CNS may use a common spatial representation of targets to plan both eye and hand movements. Evidence about this hypothesis comes primarily from studies that measure correlations between the end positions of the hand and eye (when targets are extinguished at movement onset to eliminate confounding effects of online control of hand position). One class of studies measures correlations between variable errors in eye and hand endpoints for simple pointing movements. These studies have found only weak correlations between the variable errors (Biguer et al., 1984; Prablanc et al., 1979); however, it should be noted that independent motor noise in the two effector systems will reduce correlations between the endpoints. Another class of studies follows a similar logic but uses illusory stimulus configurations (e.g., the Muller–Lyer illusion) to induce large variations in the perceived extent or positions of targets that are physically equivalent. These studies have found that perceptual changes induced by changes in stimulus configurations (e.g., changing the arrows in a Muller–Lyer figure from inward to outward pointing) have similar and correlated effects on eye and hand movement endpoints, again, when the target is extinguished on movement onset (Binsted et al., 2001; Binsted & Elliott, 1999; de Grave et al., 2006; Mack et al., 1985). The difficulty with interpreting these results is that, while they suggest that eye and hand movement planning rely on common processing systems at some stage of the visual pathway, they do not necessarily imply that a common perceptual representation of target location (or movement extent) serves as input to both. 

The current results show that spatial information in VSTM contributes more strongly to hand movement plans than to saccade plans, suggesting that the two effector systems are driven by separate spatial representations of the target. Dissociations like those found in the current experiment are not in conflict with earlier results showing correlated effects of illusory figures on eye and hand movements but rather indicate something about where in sensory processing spatial information diverges into two sensorimotor processing streams. In the current experiment, nothing about the results precludes the possibility that a single representation of the visual target is common to both effector systems just prior to integration with information in VSTM. 

Neuroanatomical considerations

Distinct areas in PPC of macaques represent visual targets for saccades (area LIP; Andersen, Snyder et al., 1998; Snyder, Batista et al., 1997, 2000) and for reaches (area MIP, the so-called parietal reach region; Andersen et al., 1998; Snyder et al., 1997, 2000). Neurons in both areas show initial sensory responses tuned to the spatial location of a target (invariant to which effector will be used to target a stimulus), followed by delay-period responses that are modulated by which effector is used to move to the target (eye or hand; Snyder, Batista et al., 1998); thus, these areas appear to be involved in the transformation from sensory information about the spatial location of a stimulus to the appropriate motor commands to target the stimulus. Similar functionally distinct regions have been found in human parietal cortex (Vesia, Prime et al., 2011; Vesia, Yan et al., 2008). 

In both saccade- and reach-related areas, the initial and delay-period responses code stimulus location in fixation-centered coordinates (Batista, Buneo et al., 1999; Medendorp, Goltz et al., 2003). Most relevant to the current study is the finding that areas coding target location for both saccades and reaching movements in PPC code the remembered locations of targets in fixation-centered coordinates even after intervening saccades change the remembered location in fixation-centered coordinates (Andersen, Bracewell et al., 1990; Batista et al., 1999; Medendorp et al., 2003). This is most surprising for areas involved in hand movement planning; however, the neurophysiological data are supported by behavioral studies showing that when a hand movement to a remembered target is preceded by a saccade away from the target, the spatial location of the target is remapped into a reference frame aligned with the new point of fixation (Henriques & Crawford, 2000; Henriques, Klier et al., 1998; Henriques, Medendorp et al., 2002; Pouget, Ducom et al., 2002; Schlicht & Schrater, 2007). 

Because areas in PPC that code target location for saccades and hand movements do so in gaze-centered coordinates even after an intervening saccade to a different location, they seem promising as areas in which remembered target information is integrated with incoming visual target information. If this is the case, the anatomical dissociation makes plausible the dissociation found here between the influence of remembered target location information on saccades and reaching movements. The fact that cells in both saccade-related regions like LIP and the parietal reach region code remembered information in gaze-centered coordinates, however, raises the question of why the influence of remembered location should, from a computational perspective, differ across the two subsystems. One possible clue to an answer lies in the observation that while saccades are planned in “pure” eye-centered coordinates (i.e., in terms of visual angle), reaches require plans in external space; thus, for example, while both hand and target positions may be represented in a gaze or fixation-centered coordinate frame for purposes of computing a hand movement displacement vector (Buneo, Jarvis et al., 2002), the representation must code the depth of both and not just their monocular direction in eye-centered coordinates. This suggests that the spatial representations used for planning saccades and reaches, while both fixation-centered, may not be equivalent. If this is the case, uncertainty in the sensory signals needed to transform visual signals from a retinotopic reference frame to an intermediate, fixation-centered reference frame would add to the uncertainty of the spatial representation of targets for hand movements, accounting for the greater influence of remembered target location information on hand movement planning (Schlicht & Schrater, 2007; Sober & Sabes, 2005). 

Supplementary Materials

Supplementary PDF - Supplementary PDF 

Acknowledgments

This work was supported by the National Institutes of Health (NIH; Grant R01EY013319 to DCK). We would like to thank Thomas Thomas and Brian McCann for programming the experiment and Leslie Richardson for running subjects. 

Commercial relationships: none. 

Corresponding author: Laurel A. Issen. 

Email: [e-mail]. 

Address: Department of Brain and Cognitive Sciences, University of Rochester, 775 Library Rd., Meliora Hall Rm. 358, Box 270268, Rochester, NY 14627, USA.