Saccadic eye movements in the dark while performing an automatized sequential high-speed sensorimotor task

Abstract

Saccades during object-related everyday tasks select visual information to guide hand movements. Nevertheless, humans can perform such a task in the dark provided it was automatized beforehand. It is largely unknown whether and how saccades are executed in this case. Recently, a long-term memory (LTM)-based direct control mode of attention during the execution of well-learned sensorimotor tasks, which predicts task-relevant saccades in the dark, was proposed (R. M. Foerster, E. Carbone, H. Koesling, & W. X. Schneider, 2011). In the present study, participants performed an automatized speed-stacking task in the dark and in the light while their eye movements were recorded. Speed stacking is a sequential high-speed sensorimotor object manipulation task. Results demonstrated that participants indeed made systematic eye movements in the dark. Saccadic scan paths and the number of fixations were highly similar across illumination conditions, while fixation rates were lower and fixation durations were longer in the dark. Importantly, the eye reached a location ahead of the hands even in the dark. Finally, neither eye–hand dynamics nor saccade accuracy correlated with hand movement durations in the dark. Results support the hypothesis of an LTM-based mode of attention selection during the execution of automatized sequential high-speed sensorimotor tasks.

Introduction

The human eye has a rather small region on the retina where visual information can be processed with high resolution, the fovea. Therefore, humans move their eyes, head, and body to bring the image of informative stimuli on the fovea. Relevant information is then extracted from the foveal region and used for the current task. In object-related actions, relevant visual information is normally used to control eye movements as well as hand movements. For instance, when grasping an object, the eyes usually reach the location where the object has to be grasped sometime before the hand (e.g., Crawford, Medendorp, & Marotta, 2004; Desmurget, Pelisson, Rossetti, & Prablanc, 1998; Droll & Hayhoe, 2007; Droll, Hayhoe, Triesch, & Sullivan, 2005; Mennie, Hayhoe, & Sullivan, 2006). When placing an object, the eyes usually reach the placement location shortly before the object is set down (e.g., Ballard et al., 1992; Droll & Hayhoe, 2007; Droll et al., 2005; Epelboim et al., 1995). The same eye–hand dynamics have been found in well-learned everyday sensorimotor tasks such as sandwich or tea making (Hayhoe, Shrivastava, Mruczek, & Pelz, 2003; Land & Hayhoe, 2001; Land, Mennie, & Rusted, 1999; Land & Tatler, 2009) and throughout the learning process of novel sequential sensorimotor tasks such as speed stacking (Foerster, Carbone, Koesling, & Schneider, 2011). Moreover, it has been shown that hand movements are executed more accurately if hand target locations have been fixated beforehand (Prablanc, Desmurget, & Gréa, 2003; Prablanc, Echallier, Komilis, & Jeannerod, 1979; Prablanc, Pélisson, & Goodale, 1986). This “eye-guides-hand” benefit may explain the robustness of the finding that the eyes lead the hands (positive eye–hand time spans) in sensorimotor tasks. 

Apart from this finding of positive eye–hand time spans, Foerster et al. (2011) have found that there are also changes of gaze characteristics during learning and automatization. Participants in the study practiced the speed-stacking task for 45 min on 14 consecutive days. Speed stacking (also known as sport stacking) consists of a fixed sequence of stacking up and down pyramids of plastic cups as fast as possible. In addition, speed stacking has fixed task elements, i.e., it has a fixed set of objects to manipulate, a fixed set of actions to perform, and a fixed sequence in which specific actions have to be executed on specific objects. To compare gaze characteristics between low and high degrees of automatization, Foerster et al. recorded participants' eye movements on the first and the last speed-stacking training day. On both training days, participants fixated on a location where a cup had to be placed shortly before the corresponding hand movement was initiated. However, on the last compared to the first training day, the eye preceded the hand by a shorter absolute time delay but by a longer time delay relative to the overall stacking trial duration. Moreover, fewer fixations were needed to perform the sensorimotor task on the last training day. Importantly, the sequence of saccades, the so-called scan paths, was highly similar between participants. Land (2007) also noticed that eye movement patterns during natural tasks are often very consistent. Task-irrelevant objects are hardly ever fixated. He suggested that participants direct their eyes according to the scripts of the activity. As speed stacking has a highly predetermined structure, the script of different participants might also be very similar, causing similar scan paths between participants. 

On the basis of the results reported in Foerster et al. (2011), we suggested that the sequence of task-driven saccades might be learned and transferred to long-term memory (LTM) during automatization of sensorimotor tasks, provided that the sequence of object-related sub-actions is fixed. According to this idea, participants should rely on a more sensory-based visual search mode of attention selection early in the learning process when no prior knowledge about objects, actions, and the action-to-object sequence is available. A sensory-based control of attention means that a visual search process is necessary for selecting the relevant environmental information and that the next saccade target location is extracted from retina-based visual information. Later in the learning process, participants should rely on a more LTM-based direct mode of control of attention. An LTM-based direct control of attention means that the next saccade target location is retrieved from LTM and that this retrieval process is guided by a stored scan path. This idea of an LTM-based mode of attention selection during the execution of an automatized sensorimotor task is tested in the present study. As a first step to examine this hypothesis, we will consider the functional role of eye movements when acting without sensory visual information, namely, in complete darkness. 

Object-related sensorimotor actions can be executed in the dark, especially if they are well practiced. However, so far the “eye-guides-hand” benefits were always measured in the light (Prablanc et al., 2003, 1979, 1986). The present study aims to clarify whether such a benefit also exists in the dark. Why is it advantageous in the light to direct the eyes to a hand target location before moving the hand? Visual information about target objects can be obtained through a fixation before moving the hand. Before grasping an object, for instance, foveal information about its size, shape, and texture can facilitate hand movement calculations. However, during a lifting or placing action, humans often fixate hand target locations that do not contain any visual object or salient feature. A reanalysis of the data reported in Foerster et al. (2011) revealed that more than 50% of all fixations were directed to locations that did not contain any such visual object or salient feature. Furthermore, visual information about target locations that do not contain any visual object might facilitate hand movement planning. Before placing an object, information about objects and surfaces close to the placement location can be sampled by prior fixations of the target location. In addition, the current position of the eyes might be used as a pointer for the hand movements in a “do-it-where-I'm-looking” manner (Ballard et al., 1992; Ballard, Hayhoe, Pook, & Rao, 1997; Flanagan, Terao, & Johansson, 2008; Neggers & Bekkering, 2001; Rosenbaum, 2010). Benefiting from visual information during fixations of hand target locations is not possible in the dark, but using the eye as a pointer for the hand movement might also be possible in the absence of visual information. Thus, saccading to hand target locations might be functional even in the dark. 

How might saccades be controlled in the dark during object-related actions? Humans have to rely on sensory input other than visual information (auditory, haptic) or on memory information (short term, long term) to specify saccade target locations in the dark. If eye movements are controlled based on auditory feedback of objects in the dark, the eyes should be directed to a location where a sound has been recently produced, e.g., where two objects have recently contacted each other. Following the same logic, if haptic feedback is used to specify saccade target locations in the dark, the eyes should be directed to a location where a hand has recently contacted an object. For placing actions, this would lead to reversed eye–hand dynamics in the dark compared to those in the light, i.e., eye movements would follow hand movements instead of preceding them. Alternatively to sensory information, short-term memory (STM) information alone without the contribution of LTM motor chunks could be used to control eye movements in the dark. Before the light is switched off, visual information might have been extracted and stored in STM. This visual STM (VSTM) information might be used to select saccade target locations in the dark. Because of the capacity limitation of the VSTM, eye movement control based on VSTM in the dark should be restricted to the first three to four saccades. In complex sensorimotor tasks such as speed stacking, object configurations are changing in conjunction with task execution, so that the visual information extracted from the first action's start configuration in the light cannot specify saccade target locations for successive actions in the dark. Finally, if Foerster et al.'s (2011) idea of an LTM-based mode of attention selection in automatized sequential sensorimotor tasks is valid, it should be possible in the dark to select saccade target locations from LTM during the execution of such tasks with fixed sub-action sequences. As a result, scan paths should be very similar when performing an automatized sensorimotor task in the light and in the dark. However, this assumption does not imply a complete LTM-driven mode of selection in the dark but allows for sensory-based corrections, e.g., by haptic or auditory feedback. Without sensory feedback, location errors of hand movements could increase along the scan path. 

To our knowledge, only Flanagan et al. (2008) have investigated where people direct their gaze when performing a sensorimotor task in the dark. In their second experiment, participants had to reach for a bar, contact a support surface, and place back the bar in the dark at their preferred speed (for a detailed task description, see also Johansson, Westling, Bäckström, & Flanagan, 2001). The experiment consisted of eight object manipulation trials in the dark, four without an obstacle and four with an obstacle. Participants viewed the scene of the task and its objects for 3 s at the beginning of each trial. After 3 s, an electric shutter was closed and a tone via earphones signaled that the object manipulation task had to be performed in the dark. Eye movements were recorded in the dark and compared with the results of a previous study (Johansson et al., 2001) with the same object manipulation task in the light. On the basis of the findings of Prablanc et al. (2003, 1979, 1986), Flanagan et al. assumed that looking to hand target locations in the dark might be useful because of well-learned eye-to-hand motor calculations, even though no visual information can be extracted. 

However, Flanagan et al. (2008) found largely unsystematic eye movements during their object manipulation task in the dark, i.e., eye movements were dissimilar in several respects between the light and dark conditions. Participants performed twice as many fixations in the light, and the size of reaching errors in the dark did not increase with the distance between saccade landing position and target location. There was only a weak link between task phases (sub-actions) and eye movements. For instance, more fixations were located close to the bar during grasping than during lifting and targeting. The dissimilarity of eye movement patterns between light and dark conditions, and the absence of a correlation between saccade and hand movement accuracy in the dark condition of Flanagan et al., might support the interpretation that saccades to hand target locations are no longer functional when no visual information is available. However, the object manipulation task analyzed in Flanagan et al. was not automatized beforehand. It was only performed eight times in the light (Johansson et al., 2001) before it had to be executed in the dark (Flanagan et al., 2008). Hence, these results cannot shed light on the hypothesis of an LTM-based direct mode of the control of attention during the execution of automatized sequential sensorimotor tasks. 

The present study investigates eye movements and their relationship to hand movements while participants executed an automatized sequential object-related sensorimotor task in the light and in the dark. The hypothesis of an LTM-based direct mode of the control of attention during automatized tasks was derived from the results with the speed-stacking task (Foerster et al., 2011). Therefore, we chose the same task for the present investigation. The key objective was to clarify whether systematic eye movements are made in the dark. More specifically, the key question was whether scan paths are similar across light and dark conditions. On the basis of our LTM-based direct mode of the control of attention, we expect similar scan paths in light and dark conditions during the execution of the automatized sequential high-speed stacking task. This assumption predicts also that the eye leads the hand in the dark. In addition, we wanted to know whether eye–hand dynamics and fixation locations in the dark are related to task performance. Only well-practiced participants took part in the study and performed the speed-stacking task as fast as possible both in the light and in the dark, while their stacking performance and their eye movements were measured. 

Methods

Participants

A total of 7 right-handed students from Bielefeld University, Germany, with a mean age of 26 years (SD = 1.38), participated in the experiment. All were highly trained in speed stacking (training is reported in Foerster et al., 2011). All participants had either normal or corrected-to-normal vision, were naive with respect to the aims of the study, and were paid for their participation. 

Apparatus

Speed-stacking equipment (cups, timer, and mat), infrared light sources, and a monocular mobile head-mounted eye tracker (SMI iView X™ HED) were used. The eye tracker features two head-mounted video cameras (one for recording the participants' right eye and one for recording the scene), infrared light source, and a dichroic mirror attached to a cycle helmet. To allow for gaze recording in the dark, the built-in infrared light source was replaced by an infrared light source with a wavelength range beyond the range visible for the human eye. This infrared light source was used to illuminate the participants' right eye. Additional infrared light sources were added to the scene, so that the SMI scene camera that monitors the participants' field of view could record the scene in the dark. Moreover, the SMI scene camera was modified by removing a built-in infrared filter. In the light, this infrared filter improves the image quality of the scene video, e.g., the brightness of colors. However, in the dark, this infrared filter would have interfered with the scene camera recording the IR-lit scene. The eye camera recorded gaze positions of the right eye at 200 Hz using an infrared video-based system. The direction of the eye relative to the head was detected by capturing the center of the pupil and the corneal reflection. The gaze position measured by the eye tracker was superimposed on the scene camera image and indicated by a red circle. The resulting gaze video was recorded at 25 Hz. Gaze position accuracy was approximately 0.5 degree of visual angle with a tracking precision below 0.1 degree of visual angle. Participants were seated in front of a table of 70-cm height, speed-stacking equipment placed on it at a distance of approximately 30 cm. The speed-stacking task was performed in an area of approximately 60 cm of width, 40 cm of height, and 30 cm of depth. The distance between the participants' eyes and the cups varied from approximately 20 cm to 50 cm during task execution. 

Gaze calibration procedure

Before the start of the actual gaze measurement, the eye tracker was calibrated with a five-point procedure in the dark. Participants were asked to sequentially fixate five 10-mm-diameter luminescent white stars on a 60-cm-wide and 40-cm-high cardboard box. One of the stars was located at the center, and each of the remaining four stars was located in one of the four corners of the box. The viewing distance of the calibration plane was 40 cm. Calibration accuracy was checked after each trial and the calibration was repeated if necessary. 

Task

A speed-stacking trial consisted of three sequences. First, a three-cup, a six-cup, and another three-cup pyramid had to be stacked up and then stacked down. Second, two 6-cup pyramids had to be stacked up and then stacked down. Third, a ten-cup pyramid had to be stacked up and then stacked down (see Movie 1). These configurations have to be stacked up and down out of 12 plastic cups. Participants had to perform the task as fast as possible. 

 

Movie 1

 

A participant performs the speed-stacking task in the light (left) and dark (right) conditions. To facilitate the comparison of eye movements across illumination conditions, the velocity of stacking in the light was dynamically adapted to the velocity of stacking in the dark.

Procedure

The experiment started with a 30-min speed-stacking warm-up phase in the light. Afterward, the light was switched off and the eye tracker was calibrated using the luminescent calibration stars. Calibration was checked, all remaining light sources were covered (control lights, computer screens, and the luminescent calibration stars), and participants had to perform the stacking task as fast as possible in the dark while their eye movements were recorded. During recording, participants were allowed to move their head and body freely. Stacking in the dark was repeated until participants had achieved at least five trials without errors (see Design section for error definition). Finally, the light was switched on again, calibration was checked and repeated if necessary, and participants stacked in the light until they achieved at least one accurate trial, while their eye movements were recorded. An experimental session lasted for approximately 1 h. Speed-stacking velocity was measured by a speed-stacking timer and stored on a laptop computer. Speed-stacking errors were annotated manually after each trial. In the dark, errors were reported by the participants. Error reports were checked based on the recorded gaze videos after the experiment. 

Analysis

The gaze videos of one dark and one light trial per participant were analyzed frame by frame. For maximum comparability, eye movements during each participant's fastest speed-stacking trial without errors within a condition were analyzed. To standardize gaze positions despite their varying absolute x and y locations within the video frames, the frame-by-frame analysis was based on the topological structure of the cup arrangement. To allow for the investigation of gaze positions depending on the temporal sequence of the speed-stacking task, the action-sequenced analysis procedure reported in Foerster et al. (2011) was used. Following this analysis procedure, the task was first divided into 44 “object-related actions” (ORAs). An ORA is an act that is performed on a particular object without interruption (Land & Hayhoe, 2001). Second, the schematic cup arrangement of each of the 44 ORAs was used for annotating the gaze positions manually. An additional advantage of the manual annotation is that fixation locations are annotated in an allocentric coordinate system irrespective of whether the eye, the head, or both have moved. An annotation tool written in JAVA facilitated the frame-by-frame annotation process. The frame-by-frame analysis of one participant stacking up a six-cup pyramid is presented as an example in Figure 1. In ORA 39, the right hand has to stack up the upper cup from the two-cup pile to the top of the ten-cup pyramid. In ORA 40, the “outer” cups have to be grasped (the right cup with the right hand and the left cup with the left hand), rotated by 180°, and tapped on the table. In ORA 41, the “outer” cups have to be used to stack down the ten-cup pyramid from top to bottom by letting the cups fall into one another. 

Figure 1

 

An example of ORA boxes for analyzing the gaze positions. The cups' starting configurations for ORAs 39 to 41 are represented in boxes and in video frames for the light (left side) and dark (right side) conditions. Each cup is illustrated as a trapezium with the long horizontal line as the open part of the cup. An additional horizontal line near the open part of a cup illustrates a pile of two cups. The boxes contain the cups' starting configuration of the present ORA and, at the same time, the end configuration of the previous ORA. The red dots represent the fixation locations of the participant in the interval between the start configuration of the present ORA and the start configuration of the successive ORA.

Based on the annotated video frame data, the number and duration of fixations, eye–hand spans (time and unit index, see below for definition), and saccade amplitudes were computed. For number of fixations, fixations were counted only once when they continued in subsequent ORAs. The eye–hand span is defined by the movement onset asynchrony between eye and hand movement given that both movements are directed to the same location in space. The eye–hand span can be measured as a time index or as a unit index (Foerster et al., 2011; Furneaux & Land, 1999). As in Foerster et al. (2011), the time index was called eye–hand time span and the unit index was called eye–hand unit span. The eye–hand time span was defined as the time interval in milliseconds between the start of a fixation and its associated hand movement. In the present study, the eye–hand time span was measured by the time interval between fixation and cup in hand, or the thumb, landing at the same location. Locations were counted as the same if fixation and cup in hand/thumb lay within half of a cup's height and width. Eye–hand time spans are positive if the eye reaches a location first and the hand follows. They are negative if the hand moves first and the fixation follows. The eye–hand unit span is defined as the number of ORAs that are accomplished between a fixation and its associated hand movement. In the present study, eye–hand unit span was measured by the number of ORAs during the interval between fixation and cup in hand, or the thumb, landing at the same location. Eye–hand unit spans are positive if the fixation is first and the hand follows. They are negative if the hand moves first and the fixation follows. Thus, the difference between the eye–hand time index and the eye–hand unit index is that the former spans time in milliseconds, while the latter spans ORAs. 

Finally, the x and y coordinates of each fixation with regard to the scene in the box were annotated with pixel accuracy. Pixel coordinates were transformed into centimeter coordinates for further analysis. Interrater reliability of two independent data scorers on x and y coordinates of four trials (light and dark trials of the two fastest participants) revealed moderate to high Pearson's correlation coefficients ranging from 0.71 to 0.99. In order to determine similarities of fixation sequences (the so-called scan paths) between the two conditions (light and dark), the action-sequenced linear distance method conducted by Foerster et al. (2011) was used. In the first step, this method calculates mean fixation locations for each participant's ORA separately for both conditions (Figure 2a). In the second step, Euclidean distance measures are calculated based on these mean fixation locations. In the present study, distances were calculated between light and dark conditions (Figure 2b) as well as between observed and randomly shuffled fixation locations of the light condition (Figure 2c). The former between-condition distance (Figure 2b) indicates scan path similarity between illumination conditions. The latter random baseline distance (Figure 2c) is used to evaluate the size of the scan path similarity between illumination conditions (analog to the method reported in 't Hart et al., 2009). 

Figure 2

 

Schematic illustration of the calculations of (a) mean fixation location, (b) between-condition distance, and (c) random baseline distance. (a) The mean fixation location is the averaged fixation location within the same ORA, subject, and condition. (b) Between-condition distance is calculated between illumination conditions and within the same ORA and subject. (c) Random baseline distance is calculated between randomly paired ORAs within the same subject in the light. Cups and fixations are symbolized as in Figure 1. Each additional horizontal line near the open part of a cup corresponds to one further cup in the pile. Averaged fixation locations of single ORAs are illustrated as black dots. Distances are illustrated as thick red lines. The figure does not depict observed fixations as it serves only for illustrative purposes.

The advantage of using action-sequenced linear distances to measure scan path similarity across illumination conditions is that fixations belonging to the same sub-action are compared independently of their index numbers. In addition, different numbers of fixations within two comparable scan paths do not reduce the similarity index as long as fixations belonging to the same sub-action are similarly located. Moreover, absolute distances are computed instead of region compliance as the string-edit method does (Brandt & Stark, 1997; Foulsham & Underwood, 2008; Levenshtein, 1966; Myers & Gray, 2010). For a detailed comparison of the action-sequenced linear distance method and the string-edit method, see Foerster et al. (2011). 

Design

The within-subject variable was illumination condition (light versus dark). The dependent variables were times and error rates of speed-stacking performance, as well as number, rate, and duration of fixations, eye–hand spans, and scan paths. The speed-stacking time was defined as the duration of a complete speed-stacking trial. We defined a speed-stacking error as cups falling or sliding down. If an error occurred, participants had to correct it before continuing. 

Results

Task performance

All trials (accurate and erroneous trials) were used for the comparison of task performance between light and dark. Stacking was significantly faster in the light (19.50 s) than in the dark (46.89 s) [t(6) = 6.25, MSE = 4.39, p < 0.01]. Mean error rates in the light (35.54%) were numerically but not significantly smaller than in the dark (50.83%) [t(6) = 1.60, MSE = 0.10, p > 0.05]. Mean stacking times and error rates for stacking in the light and in the dark are depicted in Figure 3. 

Figure 3

 

Speed-stacking performance measures. (a) Mean speed-stacking time with standard error of the means in light and dark conditions. (b) Mean error rate with standard error of the means in light and dark conditions.

Eye movements

As mentioned in the Analysis section, eye movements during each participant's fastest speed-stacking trial without errors per condition were analyzed in order to optimize comparability. 

Fixations and saccades

Overall, the mean number of fixations for a trial was 31.86. There was not even a numerical difference between the number of fixations in the light (31.86) condition and the number of fixations in the dark (31.86) condition [t(6) = 0.00, MSE = 3.96, p > 0.05], even though single subjects' mean number of fixations per trial differed between illumination conditions. On average, participants made 1.35 fixations per second and 0.72 fixation per ORA (SD = 0.21). Less than 2% of all fixations were directed at task-irrelevant points and there were no look-ahead fixations (fixating several seconds before acting, e.g., Pelz & Canosa, 2001) in either condition. The fixation rate was significantly higher in the light (1.77) than in the dark (0.93) condition [t(6) = 8.56, MSE = 0.10, p < 0.001]. The mean fixation duration was 648 ms. Substantially, shorter fixation durations occurred in the light (451 ms) than in the dark (844 ms) condition [t(6) = 2.58, MSE = 152.66, p < 0.05]. To analyze whether a similar number of fixations was made within the same ORAs, we calculated Pearson's correlation of number of fixations per ORA in the light and in the dark. The correlation reached significance [r = 0.82, p < 0.001]. Saccades had a mean length of 19.53 cm and saccade lengths did not differ significantly between light (21.87 cm) and dark (17.13 cm) conditions [t(6) = 2.13, MSE = 2.23, p > 0.05]. 

Eye–hand dynamics

Absolute eye–hand time spans did not significantly differ in light (399.32 ms) and dark (379.70 ms) conditions [t(6) = 0.24, MSE = 82.50, p > 0.05]. However, eye–hand time span depends on trial duration (see Foerster et al., 2011; Furneaux & Land, 1999), and participants performed faster in the light than in the dark. Therefore, we conducted paired t tests with relative eye–hand time spans (corrected by mean trial duration) and eye–hand unit spans as dependent variables. The relative eye–hand time span was significantly larger in the light (23.11) than in the dark (13.26) condition [t(6) = 6.08, MSE = 1.62, p < 0.01]. Accordingly, eye–hand unit spans were larger in the light (0.85) than in the dark (0.46) condition [t(6) = 3.17, MSE = 0.12, p < 0.05]. Thus, fixations were more often associated with the subsequent step (next ORA) in the light than in the dark, while fixations were more often associated with the current step (same ORA) in the dark than in the light. Importantly, eye–hand time and unit spans were positive across illumination conditions, i.e., the eyes reached a location prior to the hands even in the dark. In the light, 93.73% of the eye–hand time spans and 47.19% of the eye–hand unit spans were positive. In the dark, 82.27% of the eye–hand time spans and 41.84% of the eye–hand unit spans were positive. The lower percentage of positive eye–hand unit spans compared to relative eye–hand time spans in both illumination conditions is due to fixations with leading hand movements (positive eye–hand time spans) within the same ORA (zero eye–hand unit spans). 

Scan path similarity

To quantify scan path similarity between illumination conditions, the between-condition distance was tested against the random baseline distance (see Methods section). As a short reminder, between-condition distance was calculated in an ORA-sequenced fashion as Euclidean distance between mean fixation locations in the light and mean fixation locations in the dark. The between-condition distance (light versus dark) was 15.13 cm. For comparison, a cup is 7.5 cm wide and 9.5 cm high. The between-condition distance was significantly smaller than the random baseline distance (light observed versus light random) of 24.86 cm [t(6) = 4.14, MSE = 2.35, p < 0.01]. Thus, scan paths can be considered as highly similar across illumination conditions (see also Movie 1 andFigure 4). 

Figure 4

 

Scan paths of three different participants (top, middle, bottom) in the light (left boxes) and in the dark (right boxes) while stacking up first the three-cup pyramid on the left side, then the six-cup pyramid in the middle, and afterward the three-cup pyramid in the right side. Participants' fixations made during 10 successive ORAs (1 to 10) were superimposed on the schematic illustration of the three upstacked pyramids. Cups are illustrated as trapeziums and fixations are illustrated as red dots. Scan paths are indicated by numbers and black connection lines.

Relations between eye movements and task performance

We asked whether the functionality of eye movements in the speed-stacking task is determined by the “eye-guides-hand” benefit. In the light, hand movements are executed more accurately if hand target locations have been fixated beforehand (Prablanc et al., 2003, 1979, 1986). This benefit is either due to visual information or to the pointing function of the eye. In self-paced reaching movements, spatial movement accuracy can be easily measured and is a central aspect of task performance. In speed stacking, hand movement accuracy is difficult to operationalize. Cups have to be stacked on top of two adjacent cups. Therefore, there is a relatively broad spatial area in which cups can be placed to accomplish the task. In addition, in speed stacking, speed is much more important than spatial movement accuracy. A relevant indicator of speed is hand movement duration. Because of these considerations, we decided to investigate the relationships of eye movements to hand movement durations instead of hand movement accuracy. 

If the eye is used as a pointer for the hands in the dark, pointing to the right position should be related with task performance, that is, hand movement duration in speed stacking. Larger deviation of the saccade landing position from the hand target position necessitates hand movement corrections by haptic and proprioceptive feedback that should increase hand movement durations. Therefore, we analyzed whether the deviation of fixation locations in the dark from where participants normally fixate in the light was related to hand movement durations in the dark. A correlation analysis was performed between mean linear distances of fixation locations between illumination conditions and mean hand movement durations in the dark. If accurate pointing by the eyes was beneficial, small distances in fixation locations across illumination conditions should be correlated with short hand movement durations in the dark. This might be a hint for a spatial non-visual “eye-guides-hand” benefit in the dark. However, Pearson's correlation of between-condition linear distances with hand movement durations in the dark was not significant [r = −0.04, p = 0.44]. 

Discussion

In the present study, we asked whether systematic eye movements were made in the dark during the execution of an automatized sequential high-speed sensorimotor task—speed stacking. Our comparison condition refers to the same task in the light that already has revealed systematic patterns (Foerster et al., 2011). More specifically, we compared scan path similarity between light and dark conditions. Moreover, we wanted to know whether the eye leads the hand even in the dark and whether fixation location similarity across illumination conditions is beneficial for task performance in the dark. Based on the suggestion of an LTM-based mode of attention selection (Foerster et al., 2011), we expected similar scan paths in light and dark conditions during the execution of the automatized sequential high-speed stacking task. 

The following major results emerged. Most importantly, systematic eye movements during automatized speed stacking were not only made in the light but also in the dark. Scan paths were highly similar in light and dark conditions. All eye–hand span measures were positive in both illumination conditions. Speed-stacking trial durations were significantly faster in the light than in the dark, while error rates were only numerically smaller in the light. Finally, highly similar fixation locations across illumination conditions were not correlated with fast hand movements in the dark. 

The subsequent discussion is divided into the following sections. First, we will discuss the possible mechanisms of saccade target selection in the dark. Second, we will consider the present results with respect to the theories of automatization. Third, we report further evidence for an LTM influence on attention allocation. Fourth, a possible coupling of eye and hand movements will be discussed. Fifth, we will discuss whether systematic eye movements in the dark might be beneficial for task execution. 

Saccade target selection in the dark

The observation of positive eye–hand spans in sensorimotor tasks is a robust finding in the light (e.g., Foerster et al., 2011; Hayhoe et al., 2003; Land & Hayhoe, 2001; Land et al., 1999; Land & Tatler, 2009; Sailer, Flanagan, & Johansson, 2005). The eyes usually guide the hand movements in the light, especially in grasping and placing (e.g., Ballard et al., 1992; Crawford et al., 2004; Desmurget et al., 1998; Droll & Hayhoe, 2007; Droll et al., 2005; Epelboim et al., 1995; Land & Hayhoe, 2001; Land et al., 1999; Land & Tatler, 2009; Mennie et al., 2006). Directing the eyes to a hand target location before initiating the corresponding hand movement is advantageous for the accuracy of the hand movements (Prablanc et al., 2003, 1979, 1986). However, it was not clear whether the eyes guide the hands only if visual information is available. We found positive eye–hand spans when speed stacking had to be performed in the dark. Participants saccaded to hand target locations in the dark before moving the corresponding hand, although no visual input could have been extracted to specify the hand target location. By implication, participants must have used other sensory input or memory information to specify where to look next in the dark. 

In the speed-stacking task, sensory auditory and haptic input cannot specify hand target locations until the corresponding hand movement is completed. At the moment a cup is placed on the ground or on other cups, an auditory as well as a haptic feedback signal is generated. If sensory auditory and haptic input had been used to specify saccade targets in the dark, the eyes would have followed the hands, so that negative eye–hand spans would have been observed. The fact that the eye movement preceded the hand movement excludes the possibility that eye movements were controlled based on auditory or haptic feedback signals. Therefore, memory information must have been used to control eye movements while performing the speed-stacking task in the dark. Two memory sources are available to specify where to look next in an automatized sensorimotor task: short-term memory (STM) and long-term memory (LTM). In the present study, participants had executed the speed-stacking task for 30 min in the light prior to the first trial in the dark. In addition, participants viewed the overall start configuration of the speed-stacking task before they performed the task in the dark. Did participants use STM information alone without contribution of LTM motor chunks to specify saccade target locations in the dark condition? STM items can be stored for several seconds up to a few minutes (Atkinson & Shiffrin, 1968). The 30-min warm-up stacking phase in the light was followed by the eye tracker calibration procedure. It takes several minutes to calibrate the mobile head-mounted SMI eye tracker in the light and even more in the dark. Thus, it is very unlikely that STM information from the warm-up stacking phase was still available to control eye movements during speed stacking in the dark. In addition, the visual STM capacity spans approximately four items (Cowan, 2011). Speed stacking is a quite complex sequential sensorimotor task with 44 object-related actions (ORAs). The cup configurations in speed stacking are changing permanently, so that the visual information relevant for the next ORA is not available until its start configuration has been built up by previous ORAs. As only the start configuration of the very first ORA was viewed before the light was switched off, it is highly unlikely that participants could use STM information from this very first start configuration alone without LTM contribution to control their eye movements during speed stacking in the dark. 

Did participants use LTM information to specify the saccade target locations in the present study? Participants had automatized the sequential high-speed sensorimotor stacking task (Foerster et al., 2011) before they were asked to perform the task in the dark. A high degree of automatization can be inferred from a long-lasting prior practice of approximately 1300 trials per participant and a small increase in performance at later stages of training (see Foerster et al., 2011). As neither sensory nor STM information alone could be used to direct the eyes—prior to the hand—to hand target locations in the dark, participants must have used LTM information to control their eye movements. Importantly, not only eye–hand dynamics but also scan paths were very similar between illumination conditions. When participants performed the speed-stacking task in the dark, they seemed to retrieve successive saccade target locations—scan paths—directly from LTM. 

Given our evidence for LTM-based control of saccades, what does this imply for the control of covert attention? Previous experimental research (e.g., Deubel & Schneider, 1996; Findlay, 2009, for a summary) revealed that overt eye movements (“where to look next?”) in space depend on covert attentional processes, i.e., a saccade location in space has to be selected covertly by attention before the eye will move to that location. Therefore, not only eye movements but also covert spatial attention must have been controlled based on LTM information. In other words, task-relevant information of “where to attend next” for specifying “where to look next” (see, e.g., Schneider, 1995) must have been retrieved directly from LTM. In conclusion, the results of the present study support the idea of Foerster et al. (2011) that attentional selection in automatized sensorimotor tasks depends on LTM. 

Implications of present results for theories of automatization

The conception of automatic processing as a more LTM-based direct mode of the control of attention is opposed to traditional theories of automaticity (e.g., Schneider & Shiffrin, 1977a, 1977b) that deny the need for attention during automatic processing. Alternative concepts of automatization allow for attentional processes during the execution of automatized tasks (e.g., Logan, 1988, 1990; Neumann, 1984, 1990). Logan's (1988, 1990) instance theory of automatization assumes that attention is directed in a capacity-limited way to every sensory input. On the one hand, attention modulates the encoding of sensory input. On the other hand, attention to sensory input causes the activation of all corresponding LTM instances. According to the instance theory, acquiring LTM instances is substantial for automatization. However, requirements for the control of attention are quite similar before and after automatization because attention is used for selecting task-relevant sensory input among irrelevant input. Attentional selection of sensory input does not differentiate between automatic and non-automatic processing. Instead, the consequence of attention allocation to sensory input, namely, encoding versus encoding plus retrieval, differentiates between non-automatic and automatic processing, respectively. Neumann (1984, 1990) assumes that attention has to be allocated to relevant sensory input as well as to relevant LTM-stored content not only in automatic but also in non-automatic processing. However, processing is defined as automatic if sensory input and relevant skill procedures can be linked directly without “additional attention mechanisms” (Neumann, 1984, p. 281), and processing is defined as non-automatic if further attentional mechanisms are needed. Therefore, Neumann's conception is based on the assumption that several different attentional mechanisms exist, some of which are needed for automatic processing and some of which are additionally applied during non-automatic processing. 

Modifying Neumann's (1984, 1990) conception of automatization, we propose that automatic processing does not involve “additional attention mechanisms.” Instead, we suggest that automatization is associated with a change in the control of attention from a more sensory-based visual search mode to a more LTM-based direct mode. In agreement with Logan (1988, 1990) and Neumann (1984, 1990), we think that attentional selection is needed for task execution irrespective of the degree of automatization. Attention has to be allocated to task-relevant target locations prior to the execution of actions directed toward these target locations (Schneider, 1995; Schneider & Deubel, 2002). However, we think that in automatic processing, LTM is not only used to specify what is task relevant (e.g., the task instructions), but it also contains information about where relevant information can be found in the environment based on prior experience (see also Torralba, Oliva, Castelhano, & Henderson, 2006). Attentional control structures, the so-called “priority maps” (Bisley & Goldberg, 2010; Bundesen, Habekost, & Kyllingsbaek, 2011; Fecteau & Munoz, 2006; Wischnewski, Belardinelli, Schneider, & Steil, 2010), should receive input from this spatially structured LTM in automatic processing. A priority map is often conceptualized as a retinotopically organized map that contains representations of locations of real and imagined objects. In addition, each location codes priority, that is, the importance of attending to this location (Wolfe, 1994). Priorities are computed based on bottom-up saliency and top-down relevance (Fecteau & Munoz, 2006). The location with the highest attentional weight (priority) determines the next saccade target location (see, e.g., Carbone & Schneider, 2010). For non-automatic processing like in a standard visual search task, priorities are computed based on the external sensory input and top-down factors such as the current search template. For automatic processing like in a highly practiced stacking task, we assume that priority computation should be based on a direct spatial signal from LTM that codes where the next relevant information of the environment might be found. The location of the priority map that receives this LTM signal should be selected as the next saccade target. However, priority computations should also be influenced by external sensory input but to a much lesser degree than in non-automatic processing. In a sequential sensorimotor task, LTM information contains the sequence of task-relevant locations. Attention should be sequentially directed to the highly prioritized LTM locations generating the task-characteristic scan paths. In the dark condition of the present study, internal signals from LTM seem to have won the competition against the low-weighted auditory and haptic external signals that were also available. Covert and overt attention were, thus, directed to hand target locations before the hands have reached these locations. 

Further evidence for an LTM influence on attention allocation

The phenomenon of contextual cueing (Chun, 2000; Chun & Jiang, 1998; Olson & Chun, 2001) constitutes further evidence that LTM contributes to the allocation of attention. Participants in the studies of Chun and Jiang (1998) as well as in those of Chun (2000) had to perform a visual search task with the following design. Without participants' knowledge, a fixed set of spatially invariant target–distractor configurations was presented. After sufficient repetitions, the invariant target–distractor configurations implicitly cued the target locations within the configuration, leading to faster target detection compared to a control condition with variable spatial relations between targets and distractors. Thus, the invariant target–distractor configurations must have been learned and stored within LTM, enabling faster attention allocation to target locations during repeated trials. However, processing of visual input was still necessary to specify the target locations in this spatial contextual cueing paradigm (Chun, 2000; Chun & Jiang, 1998). As the configurations followed each other in a random order, the last configuration was no indicator for the next configuration and the target location. In a later study, Olson and Chun (2001) showed that not only an invariant spatial configuration but also an invariant temporal sequence can influence attention, demonstrating that temporal contextual cueing is also possible. After training in Olson and Chun's Experiment 1A, participants could identify a target faster when it followed an invariant sequence of stimulus durations than a sequence of random stimulus durations. Thus, the order of the target stimulus within a temporal sequence was used to predict when the target would appear. After training in Olson and Chun's Experiment 3A, participants could report a target faster when it followed an invariant spatiotemporal sequence of distractors that not only cued the time when the target would appear but also where it would be located among distractors. This spatiotemporal contextual cueing can be seen as evidence for attention being directed to a specific location at a specific point in time within a spatiotemporal sequence based on information acquired during learning. Further evidence for an LTM influence on eye movements was reported by Noton and Stark (1971a, 1971b). Their participants viewed line drawings several times under conditions that prevented peripheral vision, so that only fixated parts of the line drawings could be seen clearly. Scan paths were highly similar between initial and later presentations. The authors concluded that these scan paths were connected to the internal pattern representation of the line drawings in memory and that stimulus-related scan paths had been learned during the initial presentation (see also Land & Tatler, 2009, p. 52). The results from the contextual cueing and the scan path paradigm as well as the key findings of the present study suggest that it is possible to store target locations as whole scan paths for covert and overt attention allocation in an automatized sensorimotor task that consists of an invariant spatiotemporal sequence of sub-actions to objects. 

Type of coupling of eye and hand movements

How are eye and hand movements related during the execution of an automatized sequential sensorimotor task? Eye–hand dynamics in the light and in the dark provide an answer to this question. The absolute eye–hand time span was very similar between light and dark conditions. The relative eye–hand time span and the eye–hand unit span were significantly smaller in the dark than in the light. The eye–hand unit span revealed that fixations in the light were mainly associated with the subsequent sub-actions (next ORA), while fixations in the dark were more often associated with the current sub-action (same ORA). While the hand followed the eye with the same absolute time delay across illumination conditions, the successive eye movement must have followed the last eye movement with a longer delay in the dark. Fixation durations were, for instance, longer in the dark. The interval between successive eye movements might have been flexibly adapted to the actual task requirements of acting in the dark. The observed data pattern corresponds nicely with Ballard et al.'s (1997) “do-it-where-I'm-looking” strategy. According to this strategy, the hand target location is selected by the eye that guides the hand movement as a pointer. Calculations of hand movements based on the deictic signal of the eye might have led to the consistent eye–hand time spans across illumination conditions. However, visual feedback for online correction of hand movements was lacking in the dark, so that hand movements were slower, as indicated by the longer trial durations in the dark. The longer fixation durations may, therefore, have occurred because the eyes “waited” for the hands at the target location. Shorter eye–hand unit spans are then just a by-product of the constant interval between eye and hand movements and the elongated interval between successive fixations. 

The time between an eye movement and its corresponding hand movement has been found to be rather constant across sensorimotor tasks in the light (e.g., Hayhoe et al., 2003; Land & Hayhoe, 2001; Land et al., 1999; Land & Tatler, 2009) pointing to a tight coupling between eye and hand movements. Eye and hand movements might be coupled by a common selection of target locations for both movements (see Deubel & Schneider, 2003; Schneider, 1995, for the idea of a common spatial attentional signal for selecting eye and hand movement targets). Alternatively, eye and hand movements could be selected separately first and coupled during motor initiation. As many actions such as pointing, grasping, and placing require both eye and hand to be directed to the same location in space, a common motor target selection might be efficient. After automatization, the same spatial LTM signal could be used for eye and hand movements. In line with this idea, Land and Tatler (2009) argued that the schema control of a task initiates each ORA by activating the appropriate eye and hand movements simultaneously. A common selection mechanism would not only be useful because of the common target locations in space but also because a specific amount of time might be needed to update the hand movement parameters based on the preceding eye movement. There is evidence for common obligatory selection (e.g., Beurze, de Lange, Toni, & Medendorp, 2009; Neggers & Bekkering, 2000; Song & McPeek, 2009) as well as for separate but functionally linked selection of eye- and hand target locations (e.g., Calton, Dickinson, & Snyder, 2002; Jonikaitis & Deubel, 2011; Sailer, Eggert, Ditterich, & Sraube, 2000; Thompson & Westwood, 2007; Van Der Werf, Jensen, Fries, & Medendorp, 2010). 

Flanagan et al.'s (2008) study demonstrates nicely how eye and hand coupling can break down when eye movements are no longer necessary. However, Flanagan et al.'s object manipulation task was not automatized, so that eye–hand coupling had never been established. It seems likely that eye and hand movements are usually coupled in a flexible and task-dependent manner in natural everyday sensorimotor tasks, while it is also possible to control them separately if necessary. The type of coupling between eye and hand movement control might depend on the particular task schema control (Land & Tatler, 2009) and the degree of automatization. 

Why move the eyes systematically in the dark?

Why did participants saccade to hand target locations in the dark anyhow? Eye movements to hand target locations might be functional in the dark. Although no visual information can be extracted in the dark, saccading to a hand target location may have specific functions (Ballard et al., 1997; Flanagan et al., 2008; Neggers & Bekkering, 2001). Sensorimotor transformations from fixated locations to hand movements are well practiced and computing target positions for hand movements may be facilitated by signals from the eyeball (Flanagan et al., 2008). Thus, saccading to a hand target location may facilitate the computation of the motor command even if no visual input is available. Hand movements may be more accurate if they are calculated based on the spatial information from the eyeball. As a consequence, less correction should be necessary to reach the intended location, so that hand movements may also be faster. However, such non-visual “eye-guides-hand” benefits could neither be observed during the target-contacting task in Flanagan et al. (2008) nor during the automatized speed-stacking task in the present study. In Flanagan et al., the correlation between reach errors and the distance of saccade landing positions and target locations in the dark was not significant. In the present study, hand movement durations did not correlate with scan path similarity across illumination conditions (pointing to the right position). However, the limited variance between illumination conditions in the data set might have concealed an existing correlation. 

However, it is also possible that the task-related eye movements in the dark condition of the present study did not fulfill any function for hand movement control in the dark. Participants had learned eye and hand movements simultaneously before they had to perform the task in the dark. Afterward, they might have executed their previously automatized scan paths regardless of whether or not visual information was available. Deviation from automatized eye movements was neither instructed nor necessary to perform the task in the dark. In contrast, breaking automatization may even lead to attentional and other performance costs. In line with this reasoning, the imagery studies by Johansson, Holsanova, Dewhurst, and Holmqvist (2011) and Laeng and Teodorescu (2002) found that restricting eye movements impaired memory retrieval. Therefore, maintaining automatized but non-functional scan paths might have been efficient for speed stacking in the dark. 

A third alternative is that systematic eye movements in the dark facilitate memory retrieval of the ORA sequence of the task. In agreement with this idea, “looking at nothing” (Ferreira, Apel, & Henderson, 2008) has been found to be helpful for task execution. Laeng and Teodorescu (2002) revealed that scan path similarity between perception and imagery predicted later performance in a spatial memory task. Eye movements to “nothing” during imaging and in the dark might be functional for short-term and long-term memory retrieval. 

Summary

The present study provides new insights into eye movement control during the execution of automatized sensorimotor tasks. When performing an automatized sequential high-speed sensorimotor task in the dark, participants made systematic eye movements that resembled saccades in the light in several respects. Importantly, scan paths were very similar between light and dark conditions. The eyes reached an action-relevant location ahead of the hands even in complete darkness. However, performance was slower in the dark, resulting in longer fixation durations and smaller fixation rates in the dark than in the light. Finally, no relationship between scan path similarity and task performance was found in the dark. Results are interpreted as evidence for a more LTM-based direct mode of the control of spatial attention during the execution of automatized sequential sensorimotor tasks. It seems that eye and hand movements are learned jointly in sensorimotor tasks. Task-relevant eye and hand movement patterns are transferred to the LTM during learning and automatization. Once these patterns have been automatized, LTM signals determine directly both scan paths and hand trajectories simultaneously in subsequent task executions. If so, then automatized scan paths should be maintained and used in the dark, although no visual input can be extracted by the eyes. At the same time, visual information seems to be helpful for updating hand target locations resulting in substantially increased task duration in the dark. Other sensory sources (e.g., auditory and haptic) may be used to specify target location for the control of attention, eye, and hand movement. When spatial LTM information is imprecise, sensory information can be used to correct ongoing attention and gaze shifts. In our view, optimal performance is achieved by a combination of a sensory-based visual search mode and an LTM-based direct mode for the control of attention that is adapted to the task and its current degree of automatization. 

Acknowledgments

This research was supported by a publication fund from Bielefeld University and by grants from the Center of Excellence “Cognitive Interaction Technology” (CITEC). We would like to thank Thomas Hermann and Bettina Blaesing for their productive contributions to the speed-stacking project. Thanks are also extended to Michal Ljubljanac and Okka Risius who annotated the data for the interrater reliability as well as Frank Lehmke who wrote the JAVA annotation tool. Finally, we would like to thank two anonymous reviewers for helpful comments on an earlier version of this article. 

Commercial relationships: none. 

Corresponding authors: Rebecca M. Foerster and Werner X. Schneider. 

Emails: [e-mail]; [e-mail]. 

Address: Department of Psychology, Bielefeld University, P.O. Box 100131, D-33501 Bielefeld, Germany.