Freedom and rules in human sequential performance: A refractory period in eye-hand coordination

Abstract

Abstract

Abstract:

Abstract  In action sequences, the eyes and hands ought to be coordinated in precise ways. The mechanisms governing the architecture of encoding and action of several effectors remain unknown. Here we study hand and eye movements in a sequential task in which letters have to be typed while they move down through the screen. We observe a strict refractory period of about 200 ms between the initiation of manual and eye movements. Subjects do not initiate a saccade just after typing and do not type just after making the saccade. This refractory period is observed ubiquitously in every subject and in each step of the sequential task, even when keystrokes and saccades correspond to different items of the sequence—for instance when a subject types a letter that has been gazed at in a preceding fixation. These results extend classic findings of dual-task paradigms, of a bottleneck tightly locked to the response selection process, to unbounded serial routines. Interestingly, while the bottleneck is seemingly inevitable, better performing subjects can adopt a strategy to minimize the cost of the bottleneck, overlapping the refractory period with the encoding of the next item in the sequence.

Introduction

Many everyday tasks involve coordinated eye and hand movements. For example, when copy-typing a text we make series of fixations and saccades interspersed with hand movements for typing. One regular feature found in eye-hand coordination for a variety of tasks is that eyes are fixated ahead of the item to which the response is being made (Land & Lee, 1994; Shalom, Dagnino, & Sigman, 2011). This preview of upcoming items is an important factor in order to achieve interresponse intervals shorter than responses in isolation (Pashler, 1994b; Salthouse, 1986). Looking ahead takes advantage of the parallelism in the human cognitive architecture to overlap portions of the stimulus processing for successive actions. 

However, while the majority of brain computations and cognitive operations can proceed in parallel, dual-task experiments have reliably found serial aspects of cognitive architecture (Pashler, 1994a; Telford, 1931; Zylberberg, Dehaene, Roelfsema, & Sigman, 2011). One of the most widely studied experimental paradigms is the Psychological Refractory Period (PRP; Pashler, 1984). In the PRP when human subjects are asked to make two decisions in close temporal succession as quickly as possible, there is a systematic delay of about 200–300 ms in the time to complete the second task (Pashler, 1994a). Psychological research has associated the serial bottleneck to the response selection process and more generally to a conscious decision (Sigman & Dehaene, 2005, 2006), while neural imaging studies has localized the bottleneck in fronto-parietal networks (Dux, Ivanoff, Asplund, & Marois, 2006; Marois & Ivanoff, 2005; Tombu et al., 2011). This effect is very robust and observed even when the two tasks could be potentially performed in parallel because they are logically independent and involve different sensory modalities and motor effectors (Pashler, 1994a). The bottleneck also persists after extensive practice (Kamienkowski, Pashler, Dehaene, & Sigman, 2011; Ruthruff, Johnston, Van Selst, Whitsell, & Remington, 2003; Ruthruff, Van Selst, Johnston, & Remington, 2006) and remains unnoticed to the subject (Corallo, Sackur, Dehaene, & Sigman, 2008; Marti, Sackur, Sigman, & Dehaene, 2010). An intrinsic serial step is also present in virtually every architecture postulated to explain aspects of human rational thought, such as arithmetic or problem solving using State, Operator, And Result (SOAR) (Laird, Newell, & Rosenbloom, 1987), Adaptive Character of Thought—Rational (ACT-R) (Anderson & Lebiere, 1998), and Executive-Process/Interactive Control (EPIC) (Meyer & Kieras, 1997). These architectures rely on the concept of “productions” to factorize complex cognitive tasks into a discrete sequence of operations, which explains why humans are so slow in the execution of even relatively simple routines such as few-digit arithmetic. 

Only a handful of studies have studied the case of serial performance in which both eyes and hand ought to be coordinated to complete one single task (Pashler, 1994b; Remington, Wu, & Pashler, 2011; Wu & Remington, 2004; Wu, Remington, & Pasher, 2004). All these studies share the same protocol: Participants make separate choice responses to a succession of five stimuli spread over a wide viewing area. The main conclusion is that saccade initiation is delayed until central processing is completed. This can be seen as a psychological refractory period acting when the decision task is Task 1 and the decision to make the saccade to the subsequent stimuli is Task 2. 

Remington, Wu, and Pashler (2011) proposed a model on how manual and ocular responses are scheduled. Their model is inspired by a traditional architecture conceived to account for dual-task PRP experiments, where it has been ubiquitously found that a response selection process of each task results in a central interference bottleneck. This model makes concrete predictions about the timing of saccades and manual responses that we set out to examine here. It assumes that the task involves a cycle that iterates the following operations: a stimulus encoding (SE) stage and the selection of the two actions that are required for the task: saccade selection (SS) and response selection (RS). The SS process involves a decision of which of the several possible targets to fixate next, while RS involves the decision of which keyboard to press next. Both are decisions within a relatively large number of options. As in the classic PRP models, the architecture postulates that SE stage can overlap with stages of another cycle (each cycle here plays the role of a task in PRP experiments) and selection operations cannot overlap. Under these hypotheses, there are two alternative ways to perform a succession of trials, depending on which selection is performed first (Figure 1). In one modality, the SE phase is followed by the RS, and only then the SS occurs. These two operations do not overlap in time. We refer to this modality as “fully sequential” (Figure 1a). In the other modality, the order of the selections is inverted, and SS is performed first. After encoding the information to respond to one stimulus, subjects select where to go (SS). After making a new saccade to the new target, the keystroke selection is made concurrently with the encoding of the next stimulus. We refer to this mode as “partially overlapping” (Figure 1b). This model predicts a reduced saccade probability at the moment of the keystroke. 

Figure 1

 

Two response modalities, illustrating a model proposed by Remington et al. (2011). RS = Response Selection phase, SS = Saccade Selection phase, SE = Stimulus Encoding phase, RE = Response Execution. RS and SS last 200 ms and cannot be processed in parallel. (a) Fully sequential mode. (b) Partially overlapping mode.

The objective of this study is to investigate the precise scheduling of a task where the same operation or routine has to be executed repeatedly on sequential stimuli. The aim is to investigate whether the bottleneck that has been found ubiquitously in dual-task experiments can also be observed in self-paced sequences of executions. For this purpose we rely on three manual and ocular tasks in which subjects have to perform a sequence of keystrokes. In Experiment 1 letters have to be typed while they move down through the screen. Letters are small, which requires subjects to foveate on the letters to identify them. In Experiment 2, subjects are required to copy-type words. In these tasks, subjects spontaneously produce a well-coordinated sequence of hand and eye movements. Experiment 3 was conceived as a control task: Eye movements and keystrokes are also executed, but no explicit coordination among them is required. 

Methods

General procedure

Stimuli were presented on a 19-in. CRT monitor (1024 × 768 pixels resolution; frame rate 100 Hz). Participants were seated in front of the monitor with the head positioned on a chin rest at a distance of 50 cm from the monitor. Eye movements were recorded with a desktop-mounted, video-based eye tracker (EyeLink 1000, SR Research Ltd., Ontario, Canada) at a sampling rate of 1000 Hz. Nominal average is accuracy 0.5°, and space resolution is 0.01° root mean square (RMS), as given by the manufacturer. The participant's gaze was calibrated with a standard 13-point grid for both eyes. Two nine-point validations were run before and after each experiment. Right-eye averaged accuracies were 0.69° ± 0.25° and 0.85° ± 0.32° (mean ± SD) respectively, slightly above manufacturer's values. Saccades were automatically detected by a heuristic algorithm implemented within the Eyelink eye-tracking system. Thresholds for automatic saccade detection were 30°/s for velocity threshold and 8000°/s² for acceleration threshold. All recordings, calibrations, and validations were binocular. Only right eye data were used for the analysis. 

All experiments were implemented in Matlab (Mathworks, Natick, MA) using Psychophysics toolbox (Brainard, 1997). A QWERTY computer keyboard was used as input device. 

Participants

Participants were volunteers recruited from the general population of the University of Buenos Aires and were paid for their participation. All subjects were native speakers of Spanish who reported normal (or corrected-to-normal) vision. The main condition to participate in Experiments 1 and 2 was to be able to touch-type (type without looking at the keyboard). 

Experiment 1: Letter rain

This experiment was based on a classic game often used to train typing skills, called Letter rain. Letters appear at the top of the screen and fall vertically at constant velocity. The task of the participants was to hit the falling letters by typing them on a computer keyboard before they reach the bottom of the screen (see Figure 2 and an example in Supplementary Video S1 or in http://youtu.be/znEKxhj32-o). It is a case of continuous task with preview, and it requires good eye-hand coordination in order to perform correctly. 

Figure 2

 

A sketch of Experiment 1.

Methods

Fifteen subjects capable of touch-typing completed this task (13% females; mean age 32, range 20–43). Letters appeared at the top of the screen in a random horizontal position. Up to a maximum of 15 letters were present at any time, meaning that once passed in the initial few seconds, the rate of letter appearance was effectively controlled by the hitting speed of the player. The falling speed of the newly appearing letters set the difficulty of the task. In order to match task difficulty to participants' skills, the falling speed was selected dynamically, according to individual performance. We applied an algorithm that increased falling speed of the newly appearing letters if performance was good (hence making the task harder), and decreased it if performance was bad, in periods of 8 s. Initial falling speed was 1.5°/s. After every period of 8 s, the falling speed was updated according to the following rule: If less than 10 errors (error keystrokes plus letters lost at the bottom of the screen) were found in the previous period, the speed of the newly appearing letters was increased in steps of 0.25°/s; if 10 to 15 errors were found, the speed was maintained; if more than 15 errors were found in that period, the speed was set to decrease eight steps in the following eight 8-s periods (reducing task difficulty), and then speed was released again. This algorithm ensured that players were below but close to their individual threshold of maximum performance. Letters were 1.4° in size, with Courier New font. The portion of the screen used in this experiment was a square of 600 × 600 pixels, subtending 24° × 24°. Subjects performed a single 10-min session. 

Global performance of the participants was estimated by the number of hit letters minus the number of error keystrokes (keystrokes of letters not present in the screen), divided by the duration of the experiment. Performance is measured in hits/s. 

Multiple simultaneous keystrokes were allowed and considered separately. Keystroke time was considered at the onset of each key press. 

Results

Since this experiment was not organized into trials, the first task of our analytic strategy was to parse the continuous task into events. This was essentially a matching procedure, aiming to identify how the sequence of eye movements related to the sequence of keystrokes. 

To this aim, we first measured the distance between gaze position and the typed letter in a window of 2 s around each keystroke (Figure 3a). Negative times indicate moments prior to the keystroke. In the game, the letter disappears when it is typed (at t = 0), and hence gaze-letter distance for positive time values was calculated based on the extrapolation of letter trajectories. This distance is minimal (below 1°) in an interval approximately between 500 ms and 200 ms before the keystroke. Typical single-distance minima are about 0.4° (see Supplementary Figure S1); however, the minimum values occur at different time on each trial, producing the averaged minimum distance of 1°. Gaze-letter distance shows two sharp increases: The first one occurs about 200 ms before the keystroke, suggesting that in a substantial amount of keystrokes contributing to these distributions, subjects saccade away from the letter 200 ms before the keystroke. The second transition is observed at around 200 ms after the keystroke, suggesting the existence of a different population of events in which the saccade away of the letter is performed 200 ms after the keystroke. 

Figure 3

 

Keystroke-related eye movements. (a) Average distance between gaze and the typed letter, locked to the time of the keystroke. (b) Keystroke-related saccades, probability of finding saccades in time, relative to each keystroke. Data was centered (mean subtracted) for each individual participant before averaging. (c) Fixation onset and offset times, relative to the keystroke, considering only the fixation associated to each keystroke. Shades are standard deviations of subjects' means.

The locking between saccades and keystrokes is observed more clearly when, instead of averaging distance, we measured saccade probability directly, locked to the time of the keystroke, as shown in Figure 3b. To build this figure, we calculated for each participant the probability of finding saccades in each time bin and subtracted the mean value. Figure 3b shows the mean and standard deviation across individual curves. This shows an oscillatory pattern with peaks at −550 ms, −100 ms, and +200 ms, and with minima at −300 ms and +100 ms, indicating moments of saccade inhibition and a strong locking of keystrokes and saccades. 

Based on this relation, we next investigated the relation between the time in which a letter is gazed at and the time in which it is typed. However, since there are multiple moving targets at any time during the task, there is no obvious a priori method to determine when each letter is being gazed at. For this purpose we first parsed the eye-movement signal in foveations, defined as the intervals between two consecutive saccades. This definition includes both fixations and smooth pursuits to moving targets. Due to the moving nature of the stimuli (letters fall at constant speed), the vast majority of these foveations (95% ± 3%, mean ± SD) have a vertical downward mean velocity, corresponding to smooth pursuit. We observed typical pursuit trajectories, which had, on average, lower velocity than the moving letters. Averaged pursuit gain was 0.87 ± 0.12 (mean ± SD), in full agreement with values found in the literature (Buizza & Schmid, 1986; Meyer, Lasker, & Robinson, 1985). The focus of this experiment is not on the investigation of the precise pattern of these trajectories. Instead we collapse all the information in a scalar distance value to map foveations to gazed at letters according to the following algorithm. 

For each correctly typed letter, we assigned a foveation based on the following procedure. For any given typed letter (l) we construct the set F(l) of all the foveations f which began after the onset of l and before l was typed, i.e., l_onset < f_onset < l_type. Note that with this criterion, a foveation to l may finish before (f_offset < l_type) or after (f_offset > l_type) the letter has been typed. Every f in F(l) is a trajectory [f_x(t), f_y(t)], with t in (f_onset, f_offset). Similarly the letter l describes a trajectory [l_x(t), l_y(t)], with t in (l_onset, l_type). Note that the onset and offset of letter and foveation trajectories are not the same. To define a distance d(f, l) between f (the foveation) and l (the letter), for all candidate foveations in F we simply considered the Euclidean distance between position of the letter and of the foveation at the time T_mid, which is (f_onset + f_offset) / 2. Finally we considered arg min_f d(f, l), i.e., the foveation that minimized the distance to the letter of the candidate set F. This foveation was assigned to the letter l. Note that in this analysis, only one foveation is associated to a typed letter, explicitly excluding the possibility of multiple fixations to a single typed letter, and allowing the possibility that one foveation might be assigned to two different letters. We come back to this issue below. 

Once a map of letter-to-foveation has been established according to the previous algorithm, we can ask what the relative timing of foveation-to-keystrokes is, analyzing the onset and the offset of the foveations relative to the keystroke (Figure 3c). 

Foveation onsets correspond to the moment of the incoming saccade. They present a wide distribution centered about 500 ms before the keystroke, consistent with the distance analysis (Figure 3a). Participants tend to direct gaze to a letter about half a second before it is typed. This distribution extends all the way to time zero, indicating that in some cases, the letter is gazed as late as 50 ms before the keystroke. This occurs quite infrequently (only 1.1% ± 0.5% SD of the foveations started in the last 50 ms before the keystroke), and this short period is clearly not enough to encode the stimulus and to select and execute the response. Instead, these rare cases probably correspond to cases of multiple foveations on single letters. 

More interestingly, the distribution of offsets (that is, the time of the outgoing saccade) presents a bimodal distribution. It shows two moments about 200 ms before and after the keystroke in which the outgoing saccade probability is high, while the probability of an outgoing saccade is comparatively lower between both peaks, closer to the precise moment of keystroke. This bimodal distribution of an outgoing saccade probability relative to the time of typed letter suggests the existence of two classes of foveations whose understanding is a main drive of successive analysis. 

In one modality, subjects gaze at a letter, type it, and only then proceed (at the cost of a delay) to gaze at the next letter. This case corresponds to the peak found 200 ms after the keystroke, and we refer to this class of foveations as “fully sequential” (Figure 1a). In the other modality (the peak at −200 ms, before the keystroke) subjects gaze at a letter and, before typing it, proceed to gaze at the next letter in the sequence. We refer to this class as “partially overlapping” (Figure 1b). We categorized each foveation simply by determining whether the end of the foveation is before (overlapping) or after (sequential) the keystroke. The fact that this distribution is bimodal assures that this classification process is reliable. 

Gaze-letter distance (Figure 3a), keystroke-related saccades (Figure 3b), and the distribution of onsets and offsets of foveations relative to the keystrokes (Figure 3c) all agree to show the same feature: If a subject gazes at a letter, types it, and only then proceeds to gaze at the following letter (sequential mode), there is a moment of inhibition of about 200 ms, i.e., the probability of making a saccade immediately is considerably reduced. This is reminiscent of processing bottlenecks that have been ubiquitously observed in response selection mechanisms in simpler dual-task setups. It is possible that this refractory period of saccades at the time of the keystroke is observed exclusively in a within-target situation when both fixation and keystroke correspond to the same target. Alternatively, saccade postponement at the moment of keystroke may reflect a limitation in the eye-hand action coordination in a sequential task even when the keystroke and the saccade are targeted to different elements of the sequence. 

To disambiguate these possibilities we measured the distribution of saccades after keystrokes separately for the sequential and partially overlapping foveations. In the sequential foveations (Figure 4a) we merely reproduced our previous observation of Figure 3c, showing that if subject gazes at a letter, types, and then proceeds to the next letter, there is a 200 ms refractory period between the moment of keystroke and the saccade. More interestingly, in the partially overlapping foveations, letter N is gazed, a saccade to letter N + 1 occurs, and during this gaze letter N is typed. After this, another saccade to letter N + 2 occurs. Figure 4b shows the time locking between the keystroke to letter N and the saccade from letter N + 1 to letter N + 2 of the sequence, i.e., the relation between a keystroke and a saccade involving distinct items of the sequence. Here also the distribution shows a sharp peak at 200 ms, reflecting a saccade postponement relative to the keystroke. In order to compare the precise timing of these maxima we fitted each dataset using two Gaussian peaks N(μ, σ). The resulting mean value and standard deviation of the peaks shown in the Figure 4a and 4b were N(187 ms, 43 ms) and N(200 ms, 38 ms), respectively. This result suggests that the delay between a keystroke and the subsequent saccade is fixed, independently of the character of the items among which the saccade is executed. 

Figure 4

 

Saccades are delayed after any keystroke. (a) Offset of outgoing fixations, locked to the keystroke in on-task condition (keystroke on letter N, saccade from N to N + 1). (b) Offset of fixations, locked to the keystroke, restricted to saccades corresponding to the saccade following a partially overlapping fixation (keystroke on letter N, saccade from N + 1 to N + 2). Shades are standard deviations of subjects' means.

In the model of Figure 1 we assume that each foveation includes two or three phases: stimulus encoding, target selection for the next saccade, and response selection (in the case of the fully sequential mode). But visual information concerning the letter in question is only used during the stimulus encoding phase. To analyze this, for each foveation associated to a letter, we have measured the point at which it is closest to the letter. We found that foveations are typically closest to letters at the beginning of the foveation (69.4% ± 4.2% of foveations are closest to the letter during the first half of the foveation; see Supplementary Figure S2 for the whole distribution), in agreement with the hypothesis that the minimum distance is reached during the encoding phase, at the first portions of the foveation. This distribution is also indicative that the foveations are usually lagging behind the letter, with a pursuit gain lower than 1, typically seen in smooth pursuit tasks. 

Our results show that in the implementation of this task there is a limitation in the temporal coordination of keystrokes and saccades. This raises the issue on how individual variability in performance can be accounted. To investigate this issue and to inquire at the single-participant level the existence of a refractory period, we measured, as done above, the distributions of the onset and offset of foveations relative to the moment of keystroke for each individual subject. 

The most striking result when observing all the distributions is that(a) the distribution of onsets are virtually identical (Figure 5a), and (b) the distribution of offsets are bimodal for each individual and that the counts on each mode vary widely across individuals (Figure 5b), while the timing of each mode is relatively invariant (Figure 5c). 

Figure 5

 

Response of individual participants. (a) Onset of fixations, relative to the time of keystroke. (b) Offset of the fixations. Grayscale corresponds to participant's performance: darker lines are higher performances. (c) Median onset and offset times of fixations for the two distinct modes, for each participant (sorted by performance). (d) Percentage of use of the fully sequential mode (offset fixation after keystroke) versus participant's performance. (e) Fixation durations of each participant as a function of performance, separated by mode.

Better performers (dark gray distributions in Figure 5b) show a very small fraction of positive peaks, indicating that most of the foveations belong to the partially overlapping class. Instead, bad performers (light gray distributions in Figure 5b) show many counts on the positive peak mode, indicating that many of the foveations are fully sequential, wasting 200 ms looking to a letter that has been typed. This is confirmed by a direct measure of the fraction of trials belonging to the partially overlapping class for each individual (Figure 5d): Better players use the fully sequential mode less frequently (about 5%), as compared to worse players [up to 45%; Slope = −22.0 (−29.0; −15.0) % / (hits/s), p < 0.0001, f-stat = 45.7, R² = 0.78]. 

We showed that the majority of foveations of good performers are in the overlapping mode, i.e., they saccade to the next letter before the keystroke more frequently. In addition, they also used other resources to organize the task to increase the number of hits per second. First, good performers make shorter fixations (Figure 5e). We observed a main effect of fixation class: Sequential fixations are almost twice as long as partially overlapping fixations, an effect of performance and a significant interaction between both terms [ANOVA effect of class: F(1, 26) = 40.53, p < 0.0001; effect of performance: F(1, 26) = 42.95, p < 0.0001; interaction: F(1, 26) = 7.51, p = 0.011]. 

Second, the efficacy of each foveation to produce a keystroke increases in good performers. In the algorithm used to match foveations to keystrokes, only the foveation closest in space is associated to the keystroke (and there are more foveations than keystrokes), which allows a fraction of the foveations to remain unassigned. This may correspond to double tracking of a single letter, to foveations that are not targeted to any letter, or to foveations to a letter that were not finalized with a keystroke. The percentage of foveations that are not assigned to a keystroke decreases with performance [Linear regression: Slope = −7.8 (−11.4; −4.2) % / (hits/s), p = 0.0005, f-stat = 21.6, R² = 0.63], indicating that good performers have a greater foveation-to-keystroke ratio (see Supplementary Figure S3). The efficacy of foveation-to-keystroke also may increase by associating a single foveation to more than one letter (looking at more than one letter at once). The number of foveations associated to two keystrokes increases with performance, but this effect did not reach significance (Supplementary Figure S3). This strategy is indeed economic since the typical duration of a foveation associated to two keystrokes that is longer than a foveation associated to one keystroke but shorter than two of such foveations. 

Summary

We have shown that in this task that subjects organize eye movements to perform a sequence of gazes to different letters. There is a precise locking between keystrokes and saccades, reflecting a refractory period of about 200 ms following each keystroke in which there is a significant inhibition of saccades. This is observed even when the saccade goes from a letter to another letter, none of which has just been typed, reflecting a fundamental limit in the motor coordination of the two effectors that are coordinated in this sequential task. The bimodal distribution of saccade-to-keystroke relation, which shows that both of these foveations cannot occur in close proximity, is observed in each individual subject. However, good performers have many more foveations in which the saccade precedes the keystroke (partially overlapping) than those in which, conversely, the keystroke precedes the saccade (fully sequential). The latter is less efficient since the time of saccade selection is not simultaneously used during the detection of the subsequent letter. Good performers also make a more effective execution of the task by decreasing the number of ineffective foveations (which do not correspond strictly to a keystroke). Altogether this shows how good performers show a variety of resources to organize the task more efficiently. However, they cannot overcome a limit that in this data appears to be intrinsic and robust of a refractory period of ∼200 ms of saccade initiation after a keystroke. 

In the next two experiments we investigate whether this limitation relates to an impossibility of coordinating hand and eye muscles in short time intervals or alternatively, if this limitation is related to processes in which both effectors ought to be coordinated in a sequential task. 

Experiment 2: Copy-typing

The copy-typing setup resembles Experiment 1 in that eye and hand movements need to be precisely coordinated to achieve good typing speeds in a different experimental setup. The presentation of words is static (as opposed to letters that move in the screen), fixations are directed to whole words or two syllables instead of single letters, and there is a well-trained regularity in the sequence of words. In fact, this task was in general considered much easier by all our participants. The objective of Experiment 2 is to examine whether the refractory period of ∼200 ms of saccade initiation after a keystroke is observed in an independent and different experimental setup which shares, with Experiment 1, the necessity of concerting keystrokes and saccades in a sequence. 

Methods

Nine subjects capable of touch-typingcompleted this task (11% females; mean age 31, range 20–43); all of them also participated in Experiment 1. 

The participants' task was to type as many words, as quickly and accurately possible, in one minute, while eye position was recorded. Three lines of high frequency 4- to 11-letter words were presented in the middle of the screen (letter size 0.8°). One word was highlighted at a time. The letters currently being typed were presented under these three lines. Participants were able to correct errors by pressing the backspace key. When the enter or space bar keys were pressed, the control passed to the next word, and no more corrections were allowed. At this moment, feedback was given: Correctly typed words were painted green and incorrect ones were painted red. When the last word of a screen was to be highlighted, a new screen appeared, repeating that word in the first place. Typing speed was measured as correct characters (the sum of characters in correctly typed words) divided by the time. Each participant repeated this procedure six times. The first three were used as practice, and the average of the last three was used as representative performance value for each participant. 

All the subjects who participated in Experiment 2 also completed Experiment 1, which allowed us to investigate covariations in performance (see Supplementary Figure S4). The 61% variability in Experiment 1 performance can be explained by the participants' typing skills [Slope = 0.48 (0.19; 0.77) (hits/s) / (chars/s), p = 0.004, f-stat = 14.16, R² = 0.61], which is expected (good typists perform better in Experiment 1). 

Results and discussion

The pattern of keystroke-related saccades (Figure 6c and d) was similar to the one observed in Experiment 1 (Figure 6a and b), showing oscillations prior to the keystroke, a decrease in probability just after the saccade, and a sharp peak 200 ms after the keystroke. The significant region (p < 0.001) of positive and negative peaks of keystroke-related saccades overlapped in both experiments [Experiment 1: negative peak (−14; 115) ms, positive peak (160; 255) ms; Experiment 2: negative peak (81; 112) ms, positive peak (228; 241) ms]. Some aspects of the keystroke-related saccades are different. For instance in the copy-typing experiments there is a weak rebound after the first peak, which most likely corresponds to the fact that each foveation is associated to a relatively regular sequence of keystrokes, and the peak reflects the locking of the last keystroke in the sequence to the saccade. Also the oscillations prior to the keystroke are broader, which is expected since foveations in this experiment are longer [Experiment 1: 241 ± 10 ms (mean ± SE); Experiment 2: 278 ± 14 ms (mean ± SE); unpaired two-tailed t test: t(22) = 2.16, p = 0.04]. Our aim here is not to provide a detailed analysis of these differences. Instead our main objective was to show that there is a consistent decrease of saccades after a keystroke followed by a peak at around 200 ms. This experiment demonstrates that the refractory period in saccade initiation relative to the keystroke is also observed in a different stimulus display, which shares with Experiment 1 the fact that saccades and keystrokes ought to be coordinated in a sequential task. 

Figure 6

 

Keystroke-related saccades for all three experiments, centered (mean subtracted) for each individual participant. (a, b) Experiment 1. (c, d) Experiment 2. (e, f) Experiment 3. Left panels: Individual participant responses. Right panels: Mean value across participants. Shades are standard deviations of subjects' means. Bars at the top and bottom corresponds to values significantly higher or lower (p < 0.001) than global mean value (dashed line). Inset: p value of the averaged keystroke-related saccades over the relevant period [200; 250] ms as a function of the sample size, for all three experiments.

The objective of the next experiment is to investigate whether the same pattern is also observed in a task that does not require any explicit coordination between eye movements and keystrokes. 

Experiment 3: Eye-hand coordination in a task not requiring any explicit coordination between eye movements and keystrokes

We conceived the next experiment in order to investigate whether there is a coupling in the timing of hand and ocular movements when they are not explicitly coordinated by the task. 

Methods

Seventeen subjects completed the task (24% females; mean age 30, range 24–38). We asked subjects to read a text on the screen, while tapping on a key at a self-paced constant rhythm with the index finger of the right hand. Subjects' rhythmic performance was controlled providing feedback when three successive keystrokes were either at latencies shorter than 350 ms or longer than 1000 ms. Subjects were asked to read a text (a Spanish translation of the introduction to Charles Darwin's The Origin of Species) for 15 min. The text was presented 10 lines of 54 characters at a time, using letter size 0.8° in Courier New font. Pressing the right arrow with their left hand allowed the participants to go to the next 10 lines of text. 

Results and discussion

We did not observe a significant keystroke-related saccade locking in this experiment (Figure 6e and f). This is not a matter of statistical power due to the number of participants since 17 participants performed Experiment 3 compared to nine who performed Experiment 2. As an additional check, we calculated the significance of the peak in the relevant period (200; 250) ms, as a function of sample size, for each of three experiments (see inset of Figure 6). We took five different random subsets of each sample size, performed t tests, and calculated the mean value of −log₁₀(p). The significance of both Experiments 1 and 2 grow with sample size, while Experiment 3 quickly stabilize at low values. This is expected since the effect is observed for every single subject for both Experiments 1 and 2 (Figure 6a through c). 

The absence of structure in the keystroke-related saccades of Experiment 3 suggests that there is no temporal coupling between saccades and keystrokes when both are produced in independent tasks. This result is in full agreement with the study of Sharikadze et al. (2009), who also found no interference between a saccadic eye movement and continuous periodic typing. 

General discussion

When humans try to make two simultaneous decisions, there is a systematic delay in one of them, reflecting a processing bottleneck, a phenomenon referred as the PRP (Pashler, 1994a; Telford, 1931). Pashler and colleagues demonstrated that the bottleneck can be mapped to the response selection process that proceeds in a strict serial fashion while other operations of the task can proceed in parallel (Pashler, 1984; Pashler & Johnston, 1989, 1998; Sigman & Dehaene, 2005). The main novelty of this work is to investigate the existence of a bottleneck in an extended task in which successive operations (of gazing and typing a letter) are paced by the participant. Since task configuration has a significant contribution to processing bottlenecks, as revealed for instance in task-switching paradigms (Allport, Styles, & Hsieh, 1994; Sigman & Dehaene, 2006), we reasoned that the bottleneck may vanish when a stationary sequence of operations is under the participant's control. Instead, we observed a robust refractory period of about 200 ms (consistent with the refractory period observed in dual tasks) in the initiation of a saccade relative to the previous keystroke. This observation was confirmed in an independent experiment in which participants had to copy-type a text. A third control experiment in which the task did not require an explicit coordination between keystrokes and saccades did not show this pattern, indicating that it does not constitute a motor limitation (the incapacity to drive both effectors in close succession) but instead a reflection of the architecture of the task. We note that this effect was observed in highly skilled typists indicating that, in accordance with observations in dual task experiments, this bottleneck does not vanish even after extensive practice (Kamienkowski et al., 2011; Lien, McCann, Ruthruff, & Proctor, 2005). 

We showed that the probability of finding saccades simultaneous with the keystroke is reduced, although it is clearly not zero (Figure 3c). One possibility is that it is simply noise. If the moment of the saccade is obtained with certain noise, one would expect a certain blurring of the 200 ms peak. However, this does not seem to be the case because the error in estimation of the saccade is not sufficient for this (typically saccades last 40 ms). Instead, we believe that typically keystrokes are inhibited at the moment of saccades, but it does not show a strict impossibility. The participants are able to type and saccade simultaneously, bypassing the bottleneck, but the probability is considerably reduced due to the interference of both selection processes. The situation is similar to classic PRP studies, where subjects are able to perform different procedures to solve the dual task, including a certain number of cases in which participants group both answers (Sigman & Dehaene, 2006). In PRP studies, grouping is explicitly avoided by instruction and can be detected, for instance, by observing that there is an effect of Stimulus-onset asynchrony (SOA) in the response to Target 1 (T1) (because participants wait for Target 2 (T2) to respond to both). Here we cannot control this, and the nonzero probability of finding saccades at the keystroke could reflect this. 

Previous studies have investigated processing limitations in hand-eye coordination in tasks in which eyes and hands are used as effectors to convey two different responses (Claeys et al., 1998; Sailer, Eggert, Ditterich, & Straube, 2000; Sharikadze, Cong, Staude, Deubel, & Wolf, 2009; Stuyven, Van der Goten, Vandierendonck, Claeys, & Crevits, 2000; see Huestegge, 2011) for a review). The main conclusion that can be drawn from these works is that rhythmic tapping is controlled by a separate periodic mechanism, which does not demand a cognitive stage that would interfere with saccade generation. This is in agreement with the work of Wei, Wertmann, and Sternad (2003), supporting the hypothesis that discrete and rhythmic tasks are two different control regimes, and coupling to other actions (like saccades) occur at different cognitive level. In Experiment 3 a single key is pressed repeatedly and no central processes associated with deciding which key to press are required. The central processes associated with the decisions of where to fixate next and which and when keystrokes are executed, conflict and delay each other. Instead, when keystrokes were produced in a steady sequence independent of the eye-movement task, they did not show interference. In Experiment 1, subjects made a deliberate selection of the item to be fixated next. Since there are multiple objects falling simultaneously, there cannot be a programmed sequence of fixations. A decision must be made as to which of the several possible targets to fixate. If the number of possible saccade targets is reduced, less central processes associated with deciding where to saccade are required; the model predicts a reduced interference. This is consistent with the fact that the interference in Experiment 2 is smaller than the Experiment 1. Furthermore, in tasks where the positions of sequential targets are fixed, even less interference is expected. Our findings are consistent with this hypothesis: Keystrokes and saccades interfere in Experiments 1 and 2 in which keystrokes and saccades have to be coordinated in a common program. Alternatively, the execution of the two decisions may involve another decisional process controlling the operations in a hierarchical architecture of nested decisions (Sigman & Dehaene, 2006), and the specific architecture of this hierarchy could adapt dynamically depending on whether the task required or didn't require explicit coordination between eye-movements and keystrokes. 

The model sketched in Figure 1 belongs to a general class of process control models, in which the saccade initiation is linked directly to the processing of the stimulus being fixated (Reichle, Pollatsek, Fisher, & Rayner, 1998; Williams & Pollatsek, 2007). Alternatively, global estimation accounts assume that the saccade timing is set by an interval timer, and this interval is modulated by recent processing history (Hooge & Erkelens, 1998; Nuthmann, Smith, Engbert, & Henderson, 2010). Remington et al. (2011) directly contrasted these two models, manipulating item and context difficulty, and showed that saccade initiation is determined principally by the state of processing on the currently fixated item, not by a rhythmic timer. Hybrid models such as Controlled Random-walk with Inhibition for Saccade Planning (CRISP) (Nuthmann et al., 2010) are based on interval timers, but allow cognitive processing to inhibit timer-determined saccade initiation. Our results are consistent with a hybrid timer model if the saccade initiation process can be delayed until central decision is completed, as suggested in Remington et al. (2011). 

An important prediction of the sequential model of Figure 1, which here we examined, is that independently of the selected order of saccade and keystroke selection, the second action should be postponed by the first. This is analogous to what has previously been observed in PRP experiments in which task order is freely decided by the participants (Sigman & Dehaene, 2006). The main difference is that here the two central processes belong to the same task. Our data is in good agreement with this prediction. A close look at Figure 5b shows that both peaks are clearly different in width and not necessarily symmetrically located with respect to the keystroke, yet they coincide in peaking around 200 ms. This effect was observed in two independent tasks and was relatively constant across subjects. While other parameters such as response order varied widely between good and bad performers, the postponement of one motor action by the precedent by about 200 ms was consistent across subjects (Figure 5). The characteristic time of about 200 ms is typically observed in other PRP experiments (Pashler, 1994a; Pashler & Johnston, 1998; Sigman & Dehaene, 2005), has been described in terms of the dynamics of ignition of cortical-decision circuits (Zylberberg et al., 2011), and cannot be avoided even after extensive training (Kamienkowski et al., 2011; Ruthruff et al., 2003; Ruthruff et al., 2006). All in all, we think that a likely explanation seems to be that the 200 ms reflect an intrinsic structural bottleneck in the coordination of sequential decisions (Dehaene & Sigman, 2012). 

Our results are much inline with the findings reported in the work of Pashler and colleages (Pashler, 1994b; Remington et al., 2011; Wu & Remington, 2004; Wu et al., 2004), finding that saccade initiation is delayed and sequentially locked to the manual response indicating the letter identity. Our study extends these results in several dimensions. First we show that the manual and ocular responses (which can be seen as the outcomes of two decisional processes) mutually delay each other. One aspect coincides with the observation of Pashler and colleagues: Deciding a letter identity delays saccade initiation. We also show that, conversely, when subjects make a saccade, there is a refractory period in which the response is inhibited. The refractory period is observed also when the subject is responding to a letter that had been gazed at before, indicating that this delay is not accounted by the processing time required to identify the letter after it has been gazed at. Second we show that this bottleneck is strict, found in every single subject who participated in this study, and independent of task performance. Subjects can find a way to organize the decisional processes in a more efficient architecture (Figure 1) by making the saccade before the keystroke and hence overlapping this refractory period with the encoding of the next item in the sequence. This can be seen as a sequential extension to N processes of the partially overlapping decision and encoding processes described in the PRP (Pashler, 1984). 

Supplementary Materials

pdf S1 

pdf S1 

pdf S1 

pdf S1 

pdf S1 

avi S1 

Acknowledgments

We thank Mauro Copelli and Fernando Rozenblit for useful discussions and ideas. We thank Roger W. Remington and another anonymous reviewer for detailed and constructive commentaries that greatly improved the manuscript. This work was partially supported by grants from Human Frontiers Science Program (HFSP) and by the Argentine National Research Council (Consejo Nacional de Investigaciones Científicas y Técnicas, CONICET). 

Commercial relationships: none. 

Corresponding author: Diego Edgar Shalom. 

Email: [e-mail] 

Address: Laboratory of Integrative Neuroscience, Physics Department, University of Buenos Aires, Buenos Aires, Argentina.