Parallel programming of saccades during natural scene viewing: Evidence from eye movement positions

Abstract

Abstract

Abstract:

Abstract  Previous studies have shown that saccade plans during natural scene viewing can be programmed in parallel. This evidence comes mainly from temporal indicators, i.e., fixation durations and latencies. In the current study, we asked whether eye movement positions recorded during scene viewing also reflect parallel programming of saccades. As participants viewed scenes in preparation for a memory task, their inspection of the scene was suddenly disrupted by a transition to another scene. We examined whether saccades after the transition were invariably directed immediately toward the center or were contingent on saccade onset times relative to the transition. The results, which showed a dissociation in eye movement behavior between two groups of saccades after the scene transition, supported the parallel programming account. Saccades with relatively long onset times (>100 ms) after the transition were directed immediately toward the center of the scene, probably to restart scene exploration. Saccades with short onset times (<100 ms) moved to the center only one saccade later. Our data on eye movement positions provide novel evidence of parallel programming of saccades during scene viewing. Additionally, results from the analyses of intersaccadic intervals were also consistent with the parallel programming hypothesis.

Introduction

During scene viewing, the eyes move about three times a second, executing a series of fixations, each lasting around 250–330 ms, separated by fast-moving saccades with velocities as high as 400–500°/s (for reviews, see Henderson, 2007; Henderson & Ferreira, 2004; Henderson & Hollingworth, 1998, 1999; Rayner, 1998, 2009). As a result of saccadic suppression, the extraction of visual information occurs only during periods of fixations (Matin, 1974). Although the eyes move serially from one location to the next during viewing, it has been proposed that underlying saccade plans could be made in parallel (Becker & Jürgens, 1979). 

Becker and Jürgens (1979) proposed that saccade plans could overlap in time (“parallel programming”). The evidence for the parallel programming of saccades comes mainly from temporal indicators. For example, when two saccades are separated by a very short delay (e.g., <150 ms), the interpretation is that the later saccade plan had been initiated before an earlier saccade plan was completed (Becker & Jürgens, 1979; McPeek, Skavenski, & Nakayama, 2000; Morrison, 1984; Theeuwes, Kramer, Hahn, & Irwin, 1998; Walker & McSorley, 2006). In one such study, Morrison presented a mask at the beginning of every eye fixation during a reading task. The mask was removed after some variable latency. He observed that some fixations were terminated very quickly (<150 ms) even when the mask was still in place. Such shorter-than-normal intersaccadic intervals (ISIs) have also been observed when participants made two consecutive saccades to track a target that moved across the screen in two quick, successive steps (Becker & Jürgens, 1979); when participants made a corrective saccade to a target after an erroneous saccade to a distractor (Godijn & Theeuwes, 2002; Theeuwes et al., 1998; Walker & McSorley, 2006); and when a mask was presented during a fixation as observers viewed images of real-world scenes (Henderson & Pierce, 2008; Henderson & Smith, 2009). If we accept that the minimum pause time between two saccades is about 175–200 ms (Rayner, Slowiaczek, Clifton, & Bertera, 1983; Salthouse & Ellis, 1980), the short ISIs suggested that the later saccade plan must have started before the earlier saccade plan was completed. This implies that the saccade plans had been programmed in parallel. 

Further, studies on saccadic inhibition have also provided temporal evidence consistent with the parallel programming hypothesis. When a visual change, such as a briefly presented blank scene (33 ms), was displayed during the visual search of a target in a natural scene, saccade executions after the visual change were sometimes delayed. This effect, known as saccadic inhibition, is characterized by a “dip” in the distribution of saccade onset times ∼100 ms after the visual change (Bompas & Sumner, 2011; Henderson & Pierce, 2008; Henderson & Smith, 2009; Pannasch, Schulz, & Velichkovsky, 2011; Reingold & Stampe, 1999, 2000, 2002, 2004; Stampe & Reingold, 2002). The distribution has been interpreted to reflect two types of saccades: Prior to the dip in the distribution, saccade executions were unaffected by the visual change, and after the dip, saccade executions were affected by the visual change (Pannasch & Velichkovsky, 2009; Reingold & Stampe, 1999, 2000). Studies have proposed that saccade planning is carried out in two consecutive stages: an initial, labile stage when a saccade plan can be cancelled or modified and a later, nonlabile stage when the saccade plan can no longer be changed and must be executed (Becker & Jürgens, 1979; Engbert, Longtin, & Kliegl, 2002; Engbert, Nuthmann, Richter, & Kliegl, 2005; Morrison, 1984; Nuthmann, Smith, Engbert, & Henderson, 2010; Reichle, Pollatsek, Fisher, & Rayner, 1998; Reichle, Rayner, & Pollatsek, 2003; Salvucci, 2001). The demarcation between the labile and nonlabile stages of a saccade plan has been referred to as the point of no return, PONR (e.g., Engbert et al., 2005; Nuthmann et al., 2010; Reichle et al., 1998). Nuthmann et al. (2010) have interpreted unaffected saccades to reflect cases in which the visual change occurred after saccade planning had proceeded past the PONR. Based on this interpretation, affected saccades would then reflect cases in which the visual change occurred before the PONR. This could include cases in which the earlier plan was still in the labile stage as well as cases in which a saccade plan has yet to be made active. The subsequent saccade execution could be delayed due to processes involved in cancelling any ongoing saccade, processing the new visual scene, and forming a new saccade plan. This account implies that it is possible for the later saccade plan to overlap the earlier plan; i.e., saccades can be programmed in parallel. 

Several models of saccade planning have adopted mechanisms of parallel programming (e.g., Engbert et al., 2005; Nuthmann et al., 2010; Reichle et al., 2003). For example, the CRISP model (Nuthmann et al., 2010), which predicts fixation durations during scene viewing, allows for the overlap of saccade plans. The model has been validated with empirical results from different scene-viewing paradigms, including the scene-onset delay (Henderson & Pierce, 2008; Henderson & Smith, 2009) and the mask-onset delay (Rayner, Smith, Malcolm, & Henderson, 2009) paradigms. However, it is important to note that CRISP is designed to be purely a model of fixation duration, and eye positions are not taken into account. 

During scene viewing, eye positions are not randomly placed and are influenced by low-level features in a scene (Itti, Koch, & Niebur, 1998; Koch & Ullman, 1985; Parkhurst, Law, & Niebur, 2002; Reinagel & Zador, 1999). Eye movement positions could also reflect parallel programming mechanisms during saccade planning. For example, Becker and Jürgens (1979) examined eye movements with a two-step paradigm and analyzed how the time interval between the second target step and the first saccadic response affected saccade landings on the first or the second target location. The results suggested that eye movements were influenced by how much the saccade plan to the first target location had been completed by the time the saccade plan to the second target location was initiated. If the time interval between the second target step and the first saccadic response was long (e.g., when the interval between the two target steps was short), the eyes landed directly on the second target location without stopping at the first target location. This suggests that the saccade plan to the first target had been cancelled, and the plan to the second target location, which was made later, was then executed. But, if the time interval between the second target step and the first saccadic response was short (e.g., when the interval between the two target steps was long), the eyes landed at both target locations consecutively, suggesting that the saccade plans to both target locations were executed. The parallel programming hypothesis could also account for skipped words during reading (Morrison, 1984) as well as reflexive eye movements toward distractors in attentional capture (McPeek et al., 2000; Theeuwes et al., 1998). Other studies have also shown that eye movement positions could be modified if a later saccade plan was initiated early enough before the execution of the earlier plan (e.g., Findlay & Harris, 1984; Ludwig, Mildinhall, & Gilchrist, 2007; Vergilino & Beauvillain, 2000; Vergilino-Perez & Beauvillain, 2004). For example, Vergilino-Perez and Beauvillain have found that in order to cancel an initial saccade program targeted toward a different part of a long word and issue a new saccade program to fixate a new target word the visual change must arrive at least 220 ms prior to saccade execution. 

During scene viewing, a systematic bias of eye movements toward the center of the scene has been observed consistently and appears to account for a significant portion of eye movements during both dynamic and static scene viewing (Bindemann, 2010; Dorr, Martinetz, Gegenfurtner, & Barth, 2010; Mannan, Wooding, & Ruddock, 1996; Mital, Smith, Hill, & Henderson, 2010; Parkhurst et al., 2002; Tatler, 2007; Tseng, Cameron, Munoz, & Itti, 2009). It has been suggested that the center may be an optimal location from which to extract overall scene information, such as gist, and may represent a convenient location to start scene exploration (Tatler, 2007). The center bias effect appears to be most prominent right after the presentation of a new scene with the clustering of fixations in the center observed to be stronger in the first fixations than in subsequent fixations (Bindemann, 2010; Parkhurst et al., 2002; Tatler, 2007). This suggests that after a scene transition, there is a spontaneous shift of the eyes toward the center. 

Present study

Evidence for the parallel programming of saccades has come largely from the temporal patterns of eye movements. One aspect that has not received equal attention is eye movement position. If saccade plans during scene viewing are under the influence of parallel programming, it should not only affect when the eyes move, but also where they move; i.e., eye positions over time should also reveal the underlying parallel programming processes. 

Participants viewed pairs of scenes consecutively in preparation for a memory task. We expected that following the transition from the first to the second scene a saccade plan to the center would be invoked. Based on the results from the center bias studies, one could predict that saccades would be executed immediately to the center of the scene after the scene transition. For example, once viewers notice that the scene has changed, they may shift their eyes to the center right away to restart scene exploration. The center could be a strategic location to obtain an overall impression of the new scene so as to plan where next to move the eyes. Thus, we should observe an increase in eye movements toward the center in the first saccade after the transition. We refer to this as the immediate centering account. However, an alternative account, taking into consideration the parallel programming of saccades, would predict that the execution of saccades to the center depends on whether the new scene was presented after saccade planning has progressed beyond the PONR to the nonlabile stage. If the new scene were presented after the PONR, the ongoing saccade plan would have to be first executed before a subsequent saccade to the center of the scene. Consequently, there would be an increase in saccades to the center only in the second saccade posttransition. If the new scene were presented before the PONR, a new saccade plan would be made toward the center and executed immediately. As a result, we should observe an increase in saccades to the center in the first saccade posttransition. We refer to this as the parallel programming account. We tested these two accounts in the current study. 

In previous studies, saccades before and after the dip of saccade onset times after a scene transition have been interpreted, respectively, as cases in which the visual change occurred after and before the PONR during saccade planning (Nuthmann et al., 2010). Thus, in our analyses, the minimum point of the dip in saccade onset times was used as an index to identify cases of whether saccade planning had developed past the PONR when the transition occurred. 

Method

Participants

Twelve undergraduates (seven females) were recruited from the National University of Singapore with a mean and SD age of 22.25 ± 2.90. They all had normal or corrected-to-normal vision. All participants provided signed informed consent and were paid for their participation. 

Apparatus and environment

The experiment was carried out in a dark, windowless room. The positions of the dominant eye were recorded at a sampling rate of 2000 Hz on an EyeLink 1000 system from SR Research Ltd. Saccades were identified if they exhibited deviations in eye positions greater than 0.1° of visual angle, minimum velocities of 30°/s, and minimum acceleration of 8000°/s². Head movements were minimized during the experiment using chin and forehead rests. A nine-point calibration and validation procedure was carried out before the images were presented at the beginning of the experiment. 

Stimuli were presented on a 22-in. LCD monitor (Samsung SyncMaster 2233) with an NVIDIA Quadro FX 3450/4000 SD graphics card at a viewing distance of 57 cm. The screen subtended 34.7° (horizontal) × 27.5° (vertical) of visual angle, and all stimuli were scaled to display on the full screen (resolution: 800 × 600 pixels). The refresh rate of the display monitor was set to 120 Hz. 

Stimuli

Seventy-two images of natural scenes depicting scenery, street scenes, animals, and indoor areas, such as offices and homes, were obtained from various online image databases (e.g., Flickr and Google Images). The images were converted to grayscale, followed by contrast maximization. The mean luminance of the image was mapped to a target value of 128 (on a 0–255 scale), and the luminance range of the image was scaled by a factor such that the output levels did not exceed the 0–255 range. Images were randomly paired, and the same pairs were shown to all participants. 

Procedure

Participants were briefed to encode the images on the screen as they would be tested after each trial. They were then shown a pair of natural scene images, one presented after the other, each for 4500 ms. A blank screen lasting 300 ms followed the second image, after which the participant received a forced-choice response question that tested their memory for objects in the images. Figure 1 illustrates the procedure of one trial. The blank images were luminance-matched to the scene images. Participants initiated a trial by pressing the space bar while maintaining fixation on a dot in the center of the screen. There were five practice and 36 experimental trials. None of the images was repeated. 

Figure 1

 

Procedure for one trial.

In the forced-choice memory test, participants were presented with an object cropped from either one of the images. The task was to say whether it belonged to the first or the second image. The cropped object was equally likely to come from either image. To increase the difficulty of the memory test, different-sized objects from the images were resized to around 150 × 150 pixels with the original aspect ratio preserved. 

Eye movement data was recorded from the start of the trial to the end of the blank period following the second image. Drift correction was performed before each trial. The order of the trials was randomized for each participant. The experimental session lasted about 30 min. 

Results

We limited our analyses to trials in which the scene transitioned from Image 1 to Image 2 during an eye fixation. After discarding trials in which the scene transition occurred during a saccade (13.2% of trials) and in which the transition was preceded or followed immediately by a blink (0.46% of trials), analyses were conducted on the remaining 373 trials across all participants (86.3% of trials). 

Saccade onsets relative to scene transition

Saccade onset time was calculated by measuring the time interval from a scene transition to the end of a fixation. We looked at the distribution of saccade onsets after the scene transition for fixations during which a scene transition occurred. A dip in the distribution of saccade onset times with a minimum at ∼100 ms after a visual change has been reported in several previous studies (e.g., Henderson & Pierce, 2008; Pannasch et al., 2011; Reingold & Stampe, 1999), and we replicated this phenomenon (see Figure 2). We classified saccades based on their onset times relative to the minimum point of the dip: Early-onset saccades were executed <100 ms after the transition, and late-onset saccades were executed >100 ms after the transition. Subsequent analyses were conducted on these two groups of saccades. 

Figure 2

 

Distribution of saccade onset time relative to the scene transition (25 ms bins).

Eye movement positions

The focus of our analyses was on eye movement positions after the scene transition. The immediate centering hypothesis predicts that all eye movements (i.e., from both the early-onset and late-onset groups) will be directed toward the center in the first saccade after the transition. On the other hand, the parallel programming hypothesis predicts that eye movements from the late-onset group will be directed toward the center in the first saccade, and eye movements from the early-onset group will be directed toward the center only in the second saccade. 

To examine eye movement positions, we created vector plots illustrating the sequential movement of the eyes in consecutive saccades (Figure 3) for all participants and trials. Each line represents an eye movement from one position (position n) to the next (position n + 1). The results illustrate differences in eye movement behavior between the early-onset and late-onset saccades. After the scene transition, both groups displayed centering behavior toward the center of the scene. The critical finding is a dissociation in when the centering occurred: For the late-onset group, the eyes moved immediately to the center in the first saccade after the scene transition. For the early-onset group, there was a delay of one saccade: The eyes only moved to the center in the second saccade after the transition. The differences in the centering behavior of the early-onset and late-onset groups support the parallel programming hypothesis and not the immediate centering hypothesis. 

Figure 3

 

(a) Each line in the plots represents an eye movement from eye position n to n + 1 (indicated by a circle). (b) Vector plots of consecutive saccades. Locations within each box represent the actual screen coordinates.

To quantify the centering behavior of the eyes toward the center, we computed the deviation angle, i.e., the extent to which the direction of a saccade deviated from a saccade moving to the center. This was done by computing the absolute angular difference between two vectors: one vector from position n to n + 1 (i.e., start to end of a saccade) and another vector from position n to the center of the image. A deviation of 0° would represent a saccade toward the center of the image, and 180° would represent a saccade away from the center (Figure 4a). The frequency distribution of the deviation is plotted in Figure 4b. We observed an increase in the frequency of eye movements with smaller deviations for the first saccade in the late-onset group and for the second saccade in the early-onset group. The increase in the frequency of lower angular deviations represented an increase of eye movements toward the center of the scene. The plots imply an increase in centering eye movements during the first saccade in the late-onset group and in the second saccade for the early-onset group. 

Figure 4

 

(a) Illustration of the deviation angle used in our calculation. (b) Frequency distribution of deviation angle. 0° represents vectors moving toward the center, and 180° represents vectors moving away from the center. (c) Plot of deviation angle. Error bars show ±1 SEM.

A formal analysis was conducted on deviation angle. Deviation angle was averaged over participant means and plotted in Figure 4c. The immediate centering account predicts a decrease in deviation angle in the first saccade for both early- and late-onset groups; however, the parallel programming account predicts a decrease in deviation angle in the first saccade for the late-onset group and in the second saccade for the early-onset group. From Figure 4c, deviation was minimum in the first saccade for the late-onset group and in the second saccade for the early-onset group. A 2 (end group: early-onset, late-onset) × 3 (saccade number: baseline¹ [last], first, and second) repeated-measures ANOVA was conducted with deviation as the dependent variable. There was a significant interaction effect of end group × saccade number, F(2, 22) = 33.27, p < .0001, η^p2 = .75. Main effects were significant for saccade number, F(2, 22) = 24.86, p < .0001, η^p2 = .69, but not end group (p = .27). For the late-onset group, deviation decreased significantly in the first saccade, baseline versus first saccade: t(11) = 9.46, p < .0001; baseline versus second saccade: t(11) = .03, p = .98. For the early-onset group, deviation decreased significantly only in the second saccade, baseline versus first saccade: t(11) = .54, p = .60; baseline versus second saccade: t(11) = 5.64, p < .0001. These results showed that centering occurred in the first saccade for the late-onset group and in the second saccade for the early-onset group, thus providing support for the parallel programming account. 

Intersaccadic intervals

In this section, we analyzed ISIs around the time of the scene transition. In our study, we refer to ISIs as the duration from the end of one saccade to the start of the subsequent one (i.e., equivalent to fixation duration). 

There have been reports that presenting a visual disruption during a fixation increases fixation duration (Pannasch, 2001; Pannasch et al., 2011; Pannasch & Velichkovsky, 2009). According to the parallel programming account, we expect to observe a similar behavior for the late-onset group but not the early-onset group. Late-onset saccades represent cases in which the transition occurred early enough for any existing saccade plan to be modified and a new saccade plan to be made based on the posttransition scene. Consequently, we expect the execution of the subsequent saccade to be delayed in time, resulting in a longer ISI when the transition was presented. However, a similar increase in ISI for the early-onset group is not expected because the early-onset saccades represent cases in which modifications to any existing saccade plan is no longer possible, and the saccade plan would be executed as is. The immediate centering account, on the other hand, predicts that all saccades would be redirected toward the center immediately after a scene transition. Thus, longer ISIs for both early-onset and late-onset groups should be observed following the scene transition. 

We plotted ISIs as a function of saccade number in Figure 5. We expected that the ISIs of the two groups would return to baseline at some point. Our data suggests that this occurred after the second saccade. 

Figure 5

 

ISIs after various saccades relative to the scene transition. Error bars show ±1 SEM.

ISIs from the baseline up to after the second saccade were entered into a 2 (end group: early-onset, late-onset) × 4 (saccade number: baseline, last, first, second) repeated-measures ANOVA. To obtain a stable baseline value, baseline ISI was computed as the average of 10 ISIs before the scene transition.² There was a significant interaction effect of end group × saccade number, F(3, 33) = 34.50, p < .0001, η^p2 = .76, as well as significant main effects of end group, F(1, 11) = 18.75, p = .001, η^p2 = .63, and saccade number, F(3, 33) = 31.41, p < .0001, η^p2 = .74. For the late-onset group, the effect of saccade number was significant, F(3, 33) = 62.71, p < .0001, η^p2 = .85. The ISI during the transition was significantly greater than baseline, baseline versus last: t(11) = −8.28, p < .0001. Subsequently, ISI returned to baseline after the first saccade, baseline versus first: t(11) = 1.92, p = .082. For the early-onset group, the effect of saccade number was also significant, F(3, 33) = 8.66, p < .0001, η^p2 = .44. ISI did not differ from baseline during the transition, baseline versus last: t(11) = −1.40, p = .19. Thereafter, ISI decreased after the first saccade, baseline versus first: t(11) = 3.34, p = .007, and returned to baseline after the second saccade, baseline versus second: t(11) = −.84, p = .42. Paired t tests were conducted with a family-wise error rate of .008. The results, which showed an increase in ISI for the late-onset but not the early-onset group during the transition, support the parallel programming account. 

Accounting for selection bias

In the current study, the scene transition occurred at a fixed time (i.e., 4500 ms) from the start of a trial, and trials were selected for analysis if the scene transition occurred during a fixation. Therefore, the ISIs may have been overestimated because of an inherent bias in the way trials were selected. Trials with longer ISIs at the point of the transition have a higher probability of being selected. Consequently, the average ISI when the transition occurred could be inflated. We call this the selection bias. 

To evaluate this selection bias, we created a simulation model of the experimental data, using a bootstrap process based on ISI and saccade duration values randomly selected from the experimental data. Only ISIs during which no scene transition occurred were selected as data for the model. Analysis of model data showed that there was a significant prolongation in ISI when a simulated scene transition occurred, implying an effect of selection bias. A description of the model and a detailed analysis of the model data can be found in Appendix A. 

Given that the selection bias had a significant effect on the prolongation of ISI when the transition occurred in the model, we tested whether the prolongation of ISI, previously observed in the experimental data, was due to the selection bias. ISIs during which a transition occurred were entered into a 2 (dataset: experiment, model) × 2 (end group: early-onset, late-onset) mixed-design ANOVA. The results showed that, in general, the ISI for the experimental data was longer than the model data, F(1, 22) = 7.83, p = .010, η^p2 = .26. This implies that the prolongation of ISI in the experimental data is not simply due to the selection bias; it is also due to the scene transition. Further, the ISI for the late-onset group was longer than the early-onset group, F(1, 22) = 127.03, p < .0001, η^p2 = .85. Although the interaction was not significant (p = .13), two-tailed, pairwise comparisons showed that ISI for the experimental data was significantly higher than the model data for the late-onset group, t(22) = 2.83, p = .01, but not for the early onset group, t(22) = 1.49, p = .15. Pairwise comparisons were conducted with the family-wise error rate of .0125. Thus, even after taking selection bias into account, the previous conclusions that the scene transition led to an increase in transition ISI for the late-onset and not the early-onset group still hold. 

Low-level local image properties

Another plausible explanation for the increase in ISI during the transition is related to low-level local image properties during fixation. For example, we would expect that saccade onsets be delayed longer in the more informative regions than in less informative regions. After a scene transition, the eyes may have stayed longer at the fixated location before moving if the same location in the new image contained interesting information to process. Based on this account, we expect informativeness to be greater for late-onset than early-onset saccades. We examined whether three different indices of informativeness, saliency (Walther & Koch, 2006), luminance, and contrast values, were different between the early-onset and late-onset groups at the fixated locations in the posttransition image when the transition occurred. Saliency, luminance, and contrast values were averaged over a window measuring 2.5° to approximate the size of the fovea. Our results showed that there was no difference in saliency, t(372) = .056, p = .95; luminance, t(372) = .53, p = .60; and contrast values, t(372) = .83, p = .21 between the early-onset and late-onset groups; thus this account was rejected. 

Focal versus ambient processing

Previous studies have identified two modes of visual processing during the free-viewing of natural scenes: an early, ambient mode characterized by short ISIs and large-amplitude saccades followed by a later, focal mode of processing characterized by longer ISIs and shorter amplitude saccades (Pannasch, Helmert, Roth, Herbold, & Walter, 2008; Velichkovsky, Joos, Helmert, & Pannasch, 2005). Based on this account, we expected the first posttransition ISI (representing the early, ambient processing phase) to be shorter than the last pretransition ISI (representing the late, focal processing phase). To test this hypothesis, ISIs were entered into a 2 × 2 repeated measures ANOVA with end group (early-onset, late-onset) and saccade number (prescene and postscene transition) as repeated factors. Overall, there was a trend that the posttransition ISI was shorter than the pretransition ISI, F(1, 11) = 4.54, p = .056, η^p2 = .38. Further, the ISI of the late-onset group was longer than the early-onset group, F(1, 11) = 6.82, p = .024, η^p2 = .38. The interaction effect did not reach significance (p = .57). Our data are consistent with the view that after the transition there was a switch to ambient processing mode. 

Discussion

In this study, we investigated whether eye movement positions during natural scene viewing were under the influence of the immediate centering account or the parallel programming account. As participants planned and executed voluntary saccades around a scene during viewing, we presented an abrupt scene transition, which was expected to trigger a new saccade plan to the center. The question was whether this would lead to immediate centering for all saccades (immediate centering account) or whether centering would be contingent on whether saccade planning had progressed beyond the PONR when the transition occurred. 

Eye movement positions

We separated saccades into an early-onset and a late-onset group, based on their posttransition saccade execution times and examined their eye movement positions. We observed that saccades from the late-onset group were redirected toward the center of the scene in the saccade immediately following the scene transition. Saccades from the early-onset group were redirected toward the center one saccade later in the second saccade after the transition. The dissociation in the behavior of the early-onset and late-onset saccades shows that not all saccades were directed to the center immediately following the scene transition, thus ruling out the immediate centering account. 

Previous studies showed a clear bimodality in the distribution of saccade onset times. The interpretation of the distribution is as follows: For saccades with onset times shorter than a “dip” in the distribution, the execution of saccades following the scene transition was unaffected by the visual change; for saccades with onset times longer than the “dip,” saccade executions following the transition were affected by the visual change (Pannasch et al., 2011; Reingold & Stampe, 1999, 2000). The distinction between the two groups of saccades lies in when the scene transition occurred with respect to saccade planning (Nuthmann et al., 2010). If saccade planning has progressed beyond the PONR, saccade execution appears not to be immediately affected; however, if saccade planning has not progressed beyond the PONR, the subsequent saccade execution is delayed. In our study, early-onset saccades, which were unaffected by the transition, demonstrated a centering behavior consistent with the idea that an ongoing saccade plan has developed beyond the PONR when the transition occurred. Saccade cancellation is no longer possible and the saccade plan had to be executed. The subsequent saccade was then made to the center. On the other hand, late-onset saccades, which were delayed by the transition, showed an immediate centering behavior, suggesting that saccade planning has yet to reach the PONR. Thus, a new plan to saccade to the center could be made and executed immediately. Our findings complement existing temporal data in support of parallel programming of saccades during scene viewing. 

Intersaccadic intervals

We observed a prolongation in ISIs when the transition was presented for the late-onset group but not for the early-onset group. This result lends further support to the parallel programming account. For late-onset saccades, the transition occurred before saccade planning had reached the PONR, and the execution of the subsequent saccade could have been delayed due to a number of plausible reasons: cancelling any ongoing saccade, processing the new visual scene, or forming a new saccade plan. As these processes take time to complete, the ISI is prolonged as a consequence. The prolongation in ISI is consistent with findings of the distractor effect (Pannasch, 2001; Pannasch et al., 2011; Pannasch & Velichkovsky, 2009) as well as findings from saccadic inhibition (Reingold & Stampe, 1999, 2000, 2002, 2004; Stampe & Reingold, 2002). For early-onset saccades, the transition occurred past the PONR during saccade planning (Nuthmann et al., 2010). Here, it is too late to cancel the saccade plan, and the ongoing saccade plan is executed. Thus, for the early-onset saccades, the ISI when the transition occurred was comparable to baseline. 

A methodological problem we identified concerns presenting the scene transition at some fixed time after the start of a trial. As a result, selection bias occurs, due to trials with longer ISIs at the point of transition having a greater probability of being selected. Although this leads to an overestimation of the ISI, the prolongation of the ISI during the transition was still present even after correcting for selection bias. 

Further, we ruled out low-level factors as an explanation for the prolongation of ISI during the transition. One might argue that saccade latency for the late-onset saccades after the transition was longer than for early-onset saccades because the fixated location was more informative. However, saliency, luminance, and contrast values at the fixated location of the posttransition scene did not differ between the early-onset and late-onset groups. Thus, the eyes did not stay longer after the transition in more informative regions than in less informative regions. This finding is consistent with previous findings that the duration of a fixation had no correlation with saliency at that fixation (Itti, 2005). 

We also observed a decrease in ISI from the last pretransition ISI to the first posttransition ISI. One interpretation is that after the scene transition, the visual system switched from a focal mode of visual processing to an ambient mode of processing (Pannasch et al., 2008; Velichkovsky et al., 2005). Indeed, this decrease was observed for both the early- and late-onset groups. However, our data on the centering of eye movements suggest that, compared to the late-onset group, the effect of the scene transition is delayed by one saccade for the early-onset group. Accordingly, the switch from ambient to focal processing mode should also occur one eye movement later for the early-onset group in the second saccade. An alternative explanation for the decrease in the first ISI for the early-onset group, in line with the parallel programming account, could be that immediately after the transition, the eyes landed in a location that had been selected from the pretransition image (recall that the saccade plan was made with reference to the pretransition image). As the transition occurred after the PONR, it would have been too late to cancel the previous saccade plan, which was executed after the transition. This landing position of the saccade could well be uninformative in the posttransition image. The visual system quickly terminated the current fixation and executed a saccade to the center, which is likely to be a more optimal location (Tatler, 2007). 

In summary, the results support the parallel programming account and not the immediate centering account. Eye movement positions show that after a scene transition, not all eye movements were redirected to the center immediately, contrary to the immediate centering account. Instead, there was a dissociation in when the eyes moved to the center in line with the parallel programming account. Our results provide novel evidence, based on eye movement positions, that saccade planning can be conducted in parallel during scene viewing. Moreover, analyses on ISIs lend further support to the parallel programming account. 

Acknowledgments

This work was supported by a grant from the A*STAR Science and Engineering Research Council's Human Factors Engineering Thematic Strategic Research Program. We thank Sebastian Pannasch and an anonymous reviewer for their helpful comments on this paper. 

Commercial relationships: none. 

Corresponding author: Esther X. W. Wu. 

Email: [email protected]. 

Address: Department of Psychology, National University of Singapore, Singapore. 

Footnotes

¹  Two one-way ANOVAs were conducted within each end group with saccade number (from the last to the 10th saccade before the scene transition) as the independent factor. For the early-onset group, the effect of saccade number was not significant (p = .097); for the late-onset group, the effect of saccade number was significant, F(9, 99) = 2.34, p = .02, η^p2 = .18. Because there was a significant effect of saccade number on the deviation for the late-onset group, we selected the deviation of the last saccade before the transition as the baseline.

Footnotes

²  Two one-way ANOVAs were conducted with saccade number (from the last ISI to the 10th ISI before the scene transition) as the independent factor. There was no effect of saccade number for each end group: early-onset, F(9, 99) = 1.55, p = .14, η^p2 = .12; late-onset, F(9, 99) = .87, p = .56, η^p2 = .07. Because ISI did not differ significantly within each end group, the baseline was computed as the average of the 10 ISIs before the scene transition to obtain a more stable value.

Footnotes

³  Two one-way analyses were conducted with saccade number (from the last to the 10th saccades before the transition) as the independent factor. The results showed no significant effect of saccade number on ISI within each end group. Thus, the baseline was computed as the average of 10 ISIs before the transition to obtain a more stable value, early-onset: F(9, 99) = 1.09, p = .38, η^p2 = .09; late-onset: F(9, 99) = 1.55, p = .14, η^p2 = .12.

Appendix A

In this study, trials were selected for analysis with the criterion that a transition occurred during an eye fixation. However, as the transition was presented at a fixed time (4500 ms) from the start of a trial, this process of selection would lead to a bias in selecting trials with longer ISIs at the time of the transition. We call this the selection bias. For example, consider the hypothetical extreme cases of two ISIs, one 10 s long and the other 1 ms long. If a transition were to be presented anytime within this 10 s, the 10 s ISI would have 100% chance of being selected, and the 1 ms ISI would have 1/10,000 chance of being selected. As a consequence of the selection bias, any prolongation of ISIs observed in the experimental data, thought to be due to the scene transition, could have been confounded with the selection bias. 

The problem is further illustrated with the following simulation exercise. We modeled the experimental data by joining ISIs and saccade durations, randomly selected from the experiment, up to an accumulated duration of 9000 ms (Figure 6a). Only ISIs and saccade durations during which a scene transition did not occur were selected from the experimental data for inclusion in the model. We repeated this for 1,500 trials. This served to mimic the pattern of fixation and saccade durations during scene viewing in the experiment. Subsequently, trials in which an ISI occurred 4500 ms after trial onset (to mimic the scene transition in the experiment) were selected for further analyses. As in our experimental analyses, trials were categorized into two groups, based on the time interval from the transition to the end of the ISI. Trials were included in the early-onset group if the time interval was less than 100 ms, and trials were included in the late-onset group if the time interval was greater than 100 ms. Given that there was no real transition in the model, we expected that the average ISI when the simulated transition occurred would not be different from the ISIs prior to the transition. However, as illustrated in Figure 6b, the ISI of the model at the time of transition was still higher than the ISIs before the transition, suggesting that the current method of selecting trials for analysis had an effect of elevating ISIs at the point of the transition. 

Figure 6

 

(a) Bootstrap model of ISIs. ISIs are represented by horizontal line segments separated by saccade durations. (b) Comparison of experiment and model data. Saccade number −1 represents the last saccade prior to the scene transition. Error bars show ±1 SEM.

To test whether the increase in ISI during the transition for the model data was significant, model ISIs were entered into a 2 (end group: early-onset, late-onset) × 2 (saccade number: baseline, transition) repeated measures ANOVA. To obtain a more stable baseline, the baseline was taken to be the average of 10 ISIs before the scene transition. The results showed significant main effects in end group, F(1, 11) = 9708.93, p < .0001, η^p2 = .99, and saccade number, F(1, 11) = 9116.51, p < .0001, η^p2 = .99, as well as in their interaction, F(1, 11) = 6419.38, p < .0001, η^p2 = .99. Subsequent paired t tests showed that ISIs during the transition increased significantly from baseline in both the early- and late-onset groups, early-onset: t(11) = 4.64, p = .001; late-onset: t(11) = 117.44, p < .0001. Two-tailed paired comparisons were conducted with a family-wise error rate of .0125. Our analysis showed that the effect of the selection bias was significant. 

In our initial analyses of ISI data that did not take the selection bias into account, we made the following conclusions about the transition ISIs (i.e., ISIs when a transition occurred): (a) the scene transition led to an increase in ISI for the late-onset (b) but not the early-onset group. Here, we ask whether these conclusions would still hold after accounting for selection bias by comparing the experimental data to the model data. 

Transition ISIs were entered into a 2 (dataset: experiment, model) × 2 (end group: early-onset, late-onset) mixed-design ANOVA. The results showed that the ISI for the experimental data was significantly longer than the model data, F(1, 22) = 7.83, p = .010, η^p2 = .26, implying that the prolongation of ISI in the experimental data could be attributed to the scene transition. Further, the ISI for the late-onset group was longer than the early-onset group, F(1, 22) = 127.03, p < .0001, η^p2 = .85. Although the interaction was not significant (p = .13), two-tailed, pairwise comparisons showed that the ISI for the experimental data was significantly higher than the model data for the late-onset group, t(22) = 2.83, p = .01, but not for the early-onset group, t(22) = 1.49, p = .15. Pairwise comparisons were conducted with the family-wise error rate of .0125. This suggests that the previous conclusions that the scene transition led to an increase in ISI for the late-onset and not the early-onset group still holds even after accounting for selection bias.