An eye fixation–related potentials analysis of the P300 potential for fixations onto a target object when exploring natural scenes

Abstract

The P300 event-related potential has been extensively studied in electroencephalography with classical paradigms that force observers to not move their eyes. This potential is classically used to infer whether a target or a task-relevant stimulus was presented. Few researches have studied this potential through more ecological paradigms where observers were able to move their eyes. In this study, we examined with an ecological paradigm and an adapted methodology the P300 potential using a visual search task that involves eye movements to actively explore natural scenes and during which eye movements and electroencephalographic activity were coregistered. Averaging the electroencephalography signal time-locked to fixation onsets, a P300 potential was observed for fixations onto the target object but not for other fixations recorded for the same visual search or for fixations recorded during the free viewing without any task. Our approach consists of using control experimental conditions with similar eye movements to ensure that the P300 potential was attributable to the fact that the observer gazed at the target rather than to other factors such as eye movement pattern (the size of the previous saccade) or the “overlap issue” between the potentials elicited by two successive fixations. We also proposed to model the time overlap issue of the potentials elicited by consecutive fixations with various durations. Our results show that the P300 potential can be studied in ecological situations without any constraint on the type of visual exploration, with some precautions in the interpretation of results due to the overlap issue.

Introduction

Visual search has been extensively studied to investigate visual processing, selective attention, and decision-making processes. During a visual search task, electroencephalography (EEG) experiments show a specific event-related potential (ERP): the P300 potential This potential corresponds to a positive component with maximal amplitude in centroparietal regions (for a review, see Polich, 2012). It is elicited when observers detect a rare stimulus among distractors (Donchin, 1981; Polich, 2012; Sutton, Braren, Zubin, & John, 1965). Such potential is associated with discrimination, categorization, selection, matching processes, and decision making (Hruby & Marsalek, 2002; Picton, 1992). The P300 potential is elicited only when participants are actively involved in the task and are asked to make a final decision on the status of the stimulus. Some studies refer to the P300 or P3 potential as the P3b potential showing another P3a potential evoked by the same visual stimulus. The P3a potential is a small positive potential that is visible between 250 and 500 ms after the stimulus onset in frontal regions; the P3a potential precedes the P3b potential (Knight, 1996; Polich, 2007). A habituation effect is reported with a decrease of the P300 amplitude during the time course of an experiment (Dandekar, Ding, Privitera, Carney, & Klein, 2012). 

Paradigms used in ERP studies limit eye movements of participants. The main reason is that eye movements and blinks create artifacts in the EEG signal. However, in real-life situations, the eyes are always moving to explore a visual scene. For few years, researchers have been interested in the corecording of eye movements and EEG signals. This corecording allows researchers to have access to the brain activity linked to fixations at specific spatial positions on the scene and specific moments. The aim of experiments with corecording is to study potentials related to fixation onsets (eye fixation–related potentials; EFRPs) as a function of the fixation location in the scene and/or according to the rank of the fixation during the exploration. With this methodology, two types of noise leading to distortions have to be corrected before the EEG analysis. First, the EEG signal has to be corrected from ocular artifacts; classically, this is done by applying an independent component analysis (ICA; Graupner, Velichkovsky, Pannasch, & Marx, 2007; Nikolaev, Nakatani, Plomp, Jurica, & van Leeuwen, 2011; Ossandón, Helo, Montefusco-Siegmund, & Maldonado, 2010). Second, due to the short interfixation intervals (IFI) between two successive fixations, possible overlaps in time might occur between the early potentials elicited by a fixation and the late potentials elicited by the previous fixation, and conversely. To our knowledge in the EFRP context, no direct correction exists. Specific precautions have to be taken to deal with this overlap issue, and the overlap effect had to be taken into account in the interpretation of EFRPs (Frey et al., 2013). 

These problems might explain why few studies have used the corecording of eye movements and EEG signals to study a visual search task while allowing eye movements. Research mainly has used artificial stimuli and/or controlled the exploration (e.g., using small cues to indicate when and where observers have to make fixations; Brouwer, Reuderink, Vincent, van Gerven, & van Erp, 2013; Healy & Smeaton, 2011; Kamienkowski, Ison, Quiroga, & Sigman, 2012). All these studies reported a differentiating activity between target and nontarget objects, emerging between 480 and 500 ms, over parieto-occipital and central regions (depending on the experiment). This potential was interpreted as a P300 potential. More recently, an experiment during which participants were asked to find a target face in images of crowds also reported a late potential associated to the P300 potential (Kaunitz et al., 2014). Observers were trained before the experimental session to make longer fixations than usually reported for visual search tasks (fixation duration longer than 500 ms). Finally, two other studies, which did not directly analyze the P300 potential, were interested in the cortical activities during the time course of a visual search task (Dias, Sajda, Dmochowski, & Parra, 2013; Körner et al., 2014). Dias et al. (2013) found precursors of neural activities that indicate the success or failure to detect targets. Körner et al. (2014) showed that the neural activity on the target fixation modulated the neural activities on the three subsequent fixations, suggesting that the cognitive process involved working memory. 

This aim of our study is twofold: (a) to replicate previous P300 potential studies using natural scenes without any constraint on eye movements and (b) to study the evolution of the P300 potential elicited by successive fixations occurring during the visual search. Three main analyses are detailed. In the first analysis, we focused on the P300 potential observed during the visual search task for the first fixation onto the target object compared with fixations recorded during a free-exploration task or the visual task for scenes without a target object. A second analysis investigated the potentials elicited by fixation and refixation of the target object. In the final analysis, potentials elicited by fixations temporally before and after the fixation in the target object were studied. Before these analyses, a model was proposed to take into account the overlap issue on potentials elicited by consecutive fixations. This model provides a new measure for comparing late potentials elicited by different fixations of interest (FOI). 

Materials and method

Participants

Thirty-nine healthy adults participated in the experiment (22 females and 17 males; age range = 20–36 years, M = 24.69 years, SD = 3.49 years). Data of five other subjects were discarded from the analysis due to technical problems in data recordings. All participants had normal or corrected-to-normal vision. They were free of any medical treatment or any neurological or psychiatric disorder, past or present. The study was approved by the local French ethics committee for noninterventional research (Comité d'Ethique pour les Recherches Non Interventionnelles) of the Pôle Grenoble Cognition¹ and conducted according to the principles expressed in the Declaration of Helsinki. All participants gave their written and informed consents prior to the experiment. 

Apparatus

Stimuli were displayed on a 20-in. ViewSonic (Walnut, CA) cathode ray tube monitor, with a resolution of 768 × 1024 pixels and a 75-Hz refresh rate, located 57 cm from participants. Scenes subtended 30° × 40° of visual angle. Eye movements were recorded using the Eyelink 1000 (SR Research, Ottawa, Ontario, Canada). Both eyes were tracked with a 1000-Hz sampling rate. The head was stabilized using a chin rest. A nine-point calibration routine was carried out at the beginning of each task and was repeated every 20 trials or when the drift correction, performed every 10 trials, reported a mean error above 0.5°. The electroencephalographic activity was continuously recorded during the experimental sessions using 32 Ag/AgCl unipolar active electrodes positioned according to the extended 10-20 system (Jasper, 1958). The right earlobe and FCz electrodes were used as reference and ground electrodes, respectively. Data were amplified using a g.GAMMAsys gtec system (g.tec Inc., Schiedlberg, Austria) and sampled to 1200 Hz. An analog bandpass filter (0.01–100 Hz) and a 50-Hz notch filter were applied online. 

Stimuli

The stimuli consisted of 240 color pictures representing indoor and outdoor scenes. Scenes were selected from several sources: the Oxford Buildings data set,² different websites, (not copyrighted), and an in-house database. 

Experimental procedure

Participants performed four 20-min sessions.³ In each session, 60 scenes were randomly displayed with the constraint that a scene was displayed only once. In this study, we analyzed only the data recorded during two of the four sessions: (a) a visual search task (VS), during which participants had to indicate whether a specified object was present (half of trials) in the scene, and (b) a free exploration task (FE), during which participants simply viewed the scenes. Task order was counterbalanced between participants. 

Each trial began with a white central fixation cross displayed for 800 to 1200 ms (Figure 1, screen 2). When the participant stabilized his or her gaze on the central fixation, a scene was displayed for 4 s (gaze contingent paradigm; Figure 1, screen 3). In the VS task, the fixation cross was preceded by a question asking whether an object was present in the scene until the participants pressed one of the mouse buttons or for 5 s maximum (Figure 1, screen 1). The question was recalled after the scene display with the two possible answers: yes or no. Participants gave their answer by pressing the mouse button corresponding to one of two proposed responses (Figure 1, screen 4). The response screen was displayed until the participant's answer. The FE task was identical to the VS task with the following exceptions: No question was presented, and no response was requested. For both tasks, each trial ended with a gray screen for 1 s (Figure 1, screen 5). The VS task was performed correctly by the participants (84.50% ± 0.20% of answers were correct). The corresponded trials were used for further analysis. 

Figure 1

 

Experimental design of one trial during a VS task. During an FE task, screens 1 and 4, which displayed the question and the answer, were not shown. The label of the screen (indicated in parentheses) was not visible to participants.

Data preprocessing

Eye movement data

Saccades were automatically detected by the Eyelink software using three thresholds: velocity (30°/s), acceleration (8000°/s²), and saccadic motion (0.15°). Fixations were detected as long as there was no saccade in progress. Only fixations with durations between 50 and 1000 ms were included in the analysis. Only the dominant eye of each participant was analyzed. 

EEG data

Using EEGlab software (Delorme & Makeig, 2004), EEG data were resampled to 1000 Hz (the eye tracker sampling rate). Eye-gaze positions and EEG signals were synchronized offline on the basis of triggers sent simultaneously on both signals during the experiment using SoftEye software (Ionescu, Guyader, & Guérin-Dugué, 2009). Before EFRPs analyses, preprocessing was applied. Noisy channels were detected by visual inspection (T7, T8, TP9, and TP10) and removed from analysis. These channels were noisy for all participants. EEG data were segmented from 500 ms before the scene onset to 4000 ms after the scene onset. Segments were visually inspected offline, and those containing muscular activities or nonphysiological artifacts were rejected. Moreover, all segments corresponding to nonvalid trials (i.e., those in which participants' eyes did not maintain fixation on the cross before the scene onset) were also excluded (see Appendix A and Supplementary Table S1, column 1). To identify and to eliminate the components of EEG signal corresponding to artifacts due to eye movements (saccades and/or blinks), an ICA (Infomax ICA, EEGlab) was conducted, which was preceded by a principal component analysis (PCA; retaining number of channels minus one component—i.e., 27). Both PCA and ICA were applied for all the valid segments per participant and per condition (39 × three separated analyses). ICA is commonly used in EFRP studies to correct eye movement artifacts in EEG signal (Frey et al., 2013; Graupner et al., 2007; Nikolaev et al., 2011; Ossandón et al., 2010). Although the ICA procedure is widely used and clearly reduces artifacts, this method has not been really established to remove all artifacts. We must always keep in mind that residual artifacts may still exist. In our study, the selection of independent components related to eye movements was not automatically executed; rather components were selected visually based on the temporal evolution of the components (for the detailed procedure, see Appendix A). The visual inspection of the EEG signal after the ICA correction confirmed that eye movement artifacts were clearly reduced. 

Fixations of interest

For EFRP analyses, fixations were off-line marked according to their location on the scene (inside or outside defined regions of interest; ROI) and their rank. 

Definition of the ROI

For the VS task, when the target object was present, the ROI corresponded to a rectangle surrounding the object (VS target; Figure 2, left). When the object was absent and for the FE task, the ROI was defined using an experimental saliency map (VS salient and FE salient; Figure 2, middle and right). An experimental saliency map was computed for each scene and task using the first 12 fixations for all participants. Note that the first fixation was not included due to central bias (Tatler, 2007; Tseng, Carmi, Cameron, Munoz, & Itti, 2009). A two-dimensional Gaussian function, with an SD of 1° centered on each fixation, was spatially added. Maps were then normalized between 0 and 1. For almost all scenes, ROI corresponded to the most salient area; however, in some cases, when the most salient area corresponded to a face or text, the ROI was chosen as the second most salient area to avoid specific processing linked to face or text (Cerf, Frady, & Koch, 2009). 

Figure 2

 

Example of scenes with their ROI marked by a pink square and their experimental saliency maps for VS target, VS salient, and FE salient. Note that for the VS target, the experimental saliency maps were computed as described in the text but were not used to select the ROI. In this case the ROI simply corresponded to the target object. We observed that the target corresponded to the most salient region of the scene.

Extraction of the FOI

Three main analyses were run to analyze specific FOI and the cortical activity elicited by these fixations. For Analysis 1, the studied FOI was the first fixation inside the ROI for the three experimental conditions: VS target, VS salient, and FE salient. 

For Analyses 2 and 3, only the VS task was considered. The FOI VS target studied in Analysis 1 was renamed Target, and the FOI VS salient was used as a control condition in the following manner. For Analysis 2, three more FOIs were defined: (a) the fixation just after the target fixation still within the target object (Target+1 ROI), (b) the first fixation outside the target (Out), and (c) the first re-fixation of the target (Target2) with the constraint to have at least one fixation outside the target. The refixation was observed in 60.10% of the trials. In fact, during a scene presentation of 4 s, participants gazed at the target object for 2.56 ± 0.09 fixations, then explored elsewhere and had time to come back to fixate the object. 

For Analysis 3, four more FOIs were defined: the two previous fixations (Target-1, Target-2) and the two subsequent fixations (Target+1, Target+2), whatever their position inside or outside the target object. A total of 72.22% of the Target+1 fixations (fixations temporally just after the target fixation) occurred in the target. 

Eye movement results and analyses

For each participant and each experimental condition, the mean rank and the median⁴ duration of FOIs were computed. The saccadic context of these FOIs was also analyzed through the median⁴ amplitude of the incoming and outgoing saccades (i.e., the saccade just before and just after the FOI). For all analyses, a repeated measures analysis of variance was used with condition (i.e., FOIs) as within-subject factors, and multiple comparisons were assessed using Bonferroni correction. 

Analysis 1

In this analysis, we compared the mean rank and mean duration of FOIs VS target, VS salient, and FE salient. The condition had an effect on the mean rank, F(2, 76) = 37.81, p < 0.001. Specifically, the mean rank was higher for VS target compared with VS salient (p < 0.001) and with FE salient (p < 0.05). It was also higher for FE salient compared with VS salient (p < 0.001; Figure 3A). Condition significantly influenced the mean fixation duration, F(2, 76) = 3.73, p < 0.05. Specifically, fixations were longer for FE salient compared with VS salient (p < 0.05; Figure 3B). 

Figure 3

 

(A) Mean fixation rank, (B) mean fixation duration, and (C) mean amplitude for incoming saccades (left) and outgoing saccades (right) for the FOIs VS target, VS salient, and FE salient.

The saccadic context of these fixations was also analyzed for both incoming and outgoing saccades. Condition significantly influenced the mean amplitude of incoming saccades, F(2, 76) = 10.74, p < 0.001 (Figure 3C, left). Specifically, the saccades were shorter for VS target and VS salient compared with FE salient (p < 0.001). Condition significantly influenced the mean amplitude of outgoing saccades, F(2, 76) = 45.33, p < 0.001 (Figure 3C, right). Specifically, saccades were shorter for VS target compared with VS salient (p < 0.001) and with FE salient (p < 0.001). They were also shorter for VS salient compared with FE salient (p < 0.01). 

Analysis 2

We analyzed the mean duration of fixations and the saccadic context (amplitude of incoming and outgoing saccades) for FOIs Target, Target+1 ROI, Out, Target2, and Control. Condition significantly influenced the mean duration of fixations, F(4, 152) = 21.19, p < 0.001 (Figure 4A). Longer fixations were observed for Target+1 ROI compared with Target, Out, Target2, and Control (p < 0.001). 

Figure 4

 

(A) Mean fixation duration and (B) mean amplitude for incoming saccades (left) and outgoing saccades (right) for the FOIs Target, Target+1 ROI, Out, Target2, and Control.

Furthermore, condition significantly influenced the mean amplitude of incoming saccades, F(4, 148) = 70.57, p < 0.001 (Figure 4B, left). Shorter incoming saccades were reported for Target+1 ROI compared with Target, Out, Target2, and Control (p < 0.001). Condition also significantly influenced the mean amplitude of outgoing saccades, F(4, 152) = 44.63, p < 0.001 (Figure 4B, right). Specifically, the amplitude of outgoing saccades was larger for Out compared with Target, Target+1 ROI, Target2, and Control (p < 0.001), and saccades were shorter for Target and Target2 compared with Control (p < 0.001). 

Analysis 3

We presented the mean fixation duration and the saccadic context (amplitude of incoming and outgoing saccades) for FOIs Target-2, Target-1, Target, Target+1 ROI, Target+1, and Target+2. Condition significantly influenced the mean duration of fixations, F(5, 190) = 33.88, p < 0.001 (Figure 5A). Specifically, fixations were shorter before the target object fixation: No difference was observed between Target-2 and Target-1, but those fixations were both shorter compared with Target (p < 0.05) and Target+1 ROI, Target+1, and Target+2 (p < 0.001). Finally, fixations were also shorter for Target compared with Target+1 ROI, Target+1, and Target+2 (p < 0.001). No difference was observed between those three fixations. 

Figure 5

 

(A) Mean fixation duration and (B) mean amplitude for the incoming saccades (left) and the outgoing saccades (right) for the FOIs Target-2, Target-1, Target, Target+1 ROI, Target+1, and Target+2.

Condition significantly influenced the mean amplitude of incoming saccades, F(5, 190) = 83.83, p < 0.001 (Figure 5B, left). Smaller saccades were reported for the fixations after the target object fixations: Smaller saccades were observed for Target+1 ROI, Target+1, and Target+2 compared with Target-2, Target-1, and Target (p < 0.001). However, the incoming saccade to Target+2 was larger compared with Target+1 and Target+1 ROI (p < 0.001). Smaller saccades were also reported for Target-2 compared with Target (p < 0.05). Condition significantly influenced the mean amplitude of the outgoing saccades, F(4, 152) = 8.79, p < 0.001 (Figure 5B, right). Smaller saccades were observed when the target was fixated: The amplitude for outgoing saccades was smaller for Target, Target+1 ROI, Target+1, and Target+2 compared with Target-2 and Target-1 (p < 0.001). Saccades were also smaller after Target compared with Target+1 and Target+2 (p < 0.001). 

EFRPs results and analyses

For the EFRPs analyses, preprocessed segments (−500 to 4000 ms) were reduced into shorter segments (−200 to 700 ms), time locked to the onsets of the FOIs. Activity recorded during the period (−200 to −100 ms) before the fixation onset was used for the baseline correction. This time window was large enough to analyze late potentials with a latency of around 300 ms. For each participant, EEG segments were averaged for each experimental condition and each electrode. Statistical analyses focused on midline electrodes (i.e., Fz, Cz, and Pz), as classically done in ERP and EFRP studies (Brouwer et al., 2013; Fornaryova Key, Dove, & Maguire, 2005; Kaunitz et al., 2014; Polich, 2012). 

Fixations selection for EFRP analyses

Some eye movement parameters, such as the amplitude of the incoming saccade or fixation durations, have been shown to influence EFRPs (Kamienkowski et al., 2012). To compare EFRPs between different experimental conditions, FOIs need to have similar context between conditions. This is necessary to conclude that observed differences are due to the experimental conditions and not to different eye movement parameters. Fixations might be selected according to different parameters (Dias et al., 2013; Kaunitz et al., 2014; Körner et al., 2014). The selection depends on the objectives of the study and remains an open question. We proposed to apply an approach similar to the one adopted by Parra, Spence, Gerson, and Sajda (2005). It is based on an ocular artifact correction followed by a fixation selection process. Fixations were selected using only one criterion (fixation duration) instead of several. Main features concern the variability of fixation durations and the saccadic context (incoming and outgoing saccades). This variability implies direct modulation of elicited potentials, indirect modulations due to averaging neural responses elicited by fixations with various durations, and indirect modulations due to the overlap issue caused by adjacent responses. However, Kaunitz et al. (2014) have shown that the amplitude of the ingoing saccade mainly influenced early potentials such as P1. Furthermore, the overlaps of adjacent responses differently impact the observed potential elicited by the FOI with different durations. This study focuses on a late potential, with latency of between 250 and 500 ms, that corresponds to the range of fixation durations (around 250 ms). Hence, the main parameter to control is the duration of the FOIs between conditions. While the IFI (sum of the durations of the current fixation and the outgoing saccade) depends mainly on the fixation duration, by controlling the fixation durations, the overlap provided by the responses on subsequent fixations is also comparable across conditions. To ensure such a control on fixation durations, we face a tradeoff that allowed a good matching between the distributions of fixation durations and the risk of removing too many fixations. For each analysis, fixation durations for all conditions were gathered to form one global distribution. For a chosen number of bins (15), the empirical distribution of fixation duration for each condition is matched to the global distribution by removing some fixations in each bin. Fixations were then uniformly randomly removed for each subject and each bin. This procedure has been applied independently for all the EFRP analyses. 

EFRP results

In the following, we present for each analysis the mean number of epochs after fixation selection and the associated EFRPs. Further information concerning eye movement parameters after the selection procedure (fixation duration distributions, IFI, and saccadic orientation) is presented in Appendix B. 

Analysis 1

Analysis 1 presented potentials elicited by FOIs VS target, VS salient, and FE salient. Consequently to the procedure for selection of fixations, the number of epochs in each condition was reduced (Table 1). The expected goal was achieved: There were no significant differences between the distributions of fixation durations for the three experimental conditions (Chi-square test, df = 16, p = 0.903). 

Table 1

 

For Analysis 1, statistics (M and SD) on the number of epochs per subject for the FOIs VS target, VS salient, and FE salient before and after matching the distributions of fixation durations.

 

In Figure 6A, the topographic maps represent the amplitude of the EEG signal on the period [0, 700 ms] from the onset of the three FOIs. For the condition VS target, a P300 potential that started around 250 ms was visible; it was maximal over the parietocentral region and had a larger amplitude around 550 ms. The grand average of this potential is represented in Figure 6B. The waveforms of these potentials were in line with what is classically observed. Nevertheless, there was a shift of the EEG signal on the whole time period (see the electrodes Fz and Cz). Our proposed model of adjacent responses (see the section titled Model for evoked potentials on adjacent fixations) provides us with a consistent hypothesis to explain this kind of global shift. 

Figure 6

 

For Analysis 1, EFRPs obtained for the FOIs VS target, VS salient, and FE salient. (A) Topographic maps between 0 and 700 ms from the fixation onset. (B) EFRPs for electrodes Fz, Cz, and Pz.

Analysis 2

Analysis 2 presented potentials elicited by FOIs Target, Target+1 ROI, Out, Target2, and Control. As for Analysis 1, the number of epochs in each condition was reduced due to the selection procedure (Table 2). The small number of fixations for Target+1 ROI and Target2—around 10 for some participants—might be explained, for example, by the fact that these observers did not always look twice at the target, consequently or not. After selection, no significant difference was observed between the distributions of fixation durations (Chi-square test, df = 32, p = 0.373). 

Table 2

 

For Analysis 2, statistics (M and SD) on the number of epochs per subject for the FOIs Target, Target+1 ROI, Out, Target2, and Control before and after matching the distributions of fixation durations.

 

Table 3

 

For Analysis 3, statistics (M and SD) on the number of epochs for the FOIs Target-2, Target-1, Target, and Control (top) and the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control (bottom) before and after matching the distributions of fixation durations.

 

The topographic maps are presented in Figure 7A for the period [0, 700 ms] from the onsets of the five FOIs. The grand average of this potential is represented in Figure 7B. The waveforms of these potentials were in line with what would be expected. Nevertheless, the waveform of the potential elicited at the Target+1 ROI fixation (see for the electrodes Fz and Cz) must be discussed, and we use the model prediction for this purpose. 

Figure 7

 

For Analysis 2, EFRPs obtained for the FOIs Target, Target+1 ROI, Out, Target2, and Control. (A) Topographic maps between 0 and 700 ms from the fixation onset. (B) EFRPs for electrodes Fz, Cz, and Pz.

Analysis 3

For the sake of clarity, Analysis 3 was divided into two parts (A, B). In the first part (A), the potentials elicited by the FOIs Target-2, Target-1, Target, and Control were compared. In the second part (B), the analysis focused on the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control. For both analyses, no significant difference was observed between the distributions of fixation durations after the selection procedure (A: Chi-square test, df = 21, p = 0.533; B: Chi-square test, df = 50, p = 0.086). 

In Figure 8A, the topographic maps represent the amplitude of the EEG signal on the period [0, 700 ms] from the onsets of the FOIs Target-2, Target-1, Target, and Control. The grand average of this potential is represented in Figure 8B. As previously, the waveform of the potentials elicited at the Target-2 fixation and Target-1 fixation must be discussed. A sharp increase at the end of the time period was observed (see the electrodes Fz and Cz). We use the model prediction for this purpose. 

Figure 8

 

EFRPs obtained for the FOIs Target-2, Target-1, Target, and Control. (A) Topographic maps between 0 and 700 ms from the fixation onset. (B) EFRPs for electrodes Fz, Cz, and Pz.

The topographic maps of the EEG signal amplitude on the period [0, 700 ms] from the onsets of the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control are presented in Figure 9A. The grand average of this potential is represented in Figure 9B. The waveforms of these potentials were in line with what would be expected, with the same observations derived from Analysis 2, concerning the potential elicited at the Target+1 ROI fixation and at the Target+1 fixation (see the electrodes Fz and Cz). 

Figure 9

 

For Analysis 3, EFRPs obtained for the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control. (A) Topographic maps between 0 and 700 ms from the fixation onset. (B) EFRPs for electrodes Fz, Cz, and Pz.

Model for evoked potentials on adjacent fixations

During the visual search task, EFRPs elicited by successive fixations overlapped each other. Table 4 shows the average and the standard deviation of the observed IFI values between successive fixations. IFI values were in the range of P300 potential latency. A qualitative observation of the evoked potentials elicited at the Target fixation (Analyses 1, 2, and 3), at the Target −1 fixation (Analysis 3), at the Target+1 ROI fixation (Analyses 2 and 3), and at the Target+1 fixation (Analysis 3) showed that theses EFRPs seemed to be biased. The objective of the proposed model is twofold: (a) to explain these observed waveforms and, consequently, (b) to propose a measure for comparing the P300 potential amplitude between different FOIs. We assumed that the bias was due to overlaps between potentials elicited by consecutive fixations. Hence, to compute the average amplitude on a given time interval—here [250, 600 ms]—comparing P300 potentials elicited by different FOIs as classically done is not suitable. Indeed, this measure integrates at least four contributions: (a) the amplitude of the potential of interest, (b) a part of the amplitude of the previous potential, (c) a part of the amplitude of the subsequent potential, and (c) the noise. This measure is appropriate if the part of three last contributions might be negligible. In order to better evaluate these contributions and to propose an alternative measure, a simple additive model is proposed to take into account overlaps of the potentials elicited by consecutive fixations. During a time window synchronized with the onset of a given FOI, the observed neural activity x_i(t) for the trial i can be modeled by  

 

where S(t), S_p(t), and S_s(t) denote the potentials elicited by the FOI, by the previous fixation, and on the subsequent fixation, respectively. The related positive IFI values are τp_i, and τs_i. n_i(t) denotes the outgoing brain activities that are not related to fixations. This equation is based on the assumption that only one fixation before and one after the FOI overlap with the potential elicited by this specific FOI. Table 4 shows the average and the standard deviation of the observed IFI values between successive fixations.  

Table 4

 

Mean IFI values and standard deviation for the four consecutive fixations on both sides of the Target FOI in the VS target condition.

 

Consequently, the Target-2 fixation occurred on average 487 ms (246 + 241) before the Target FOI. The overlap of the potential elicited by the Target-2 fixation can be neglected if this potential is not a late one with a latency greater than 400 ms. This was consistent with the observed EFRP time locked on the Target-2 in the VS target (Figure 8B). A similar reasoning applied to the Target+2 fixation. On average, this fixation occurred 568 ms (276 + 292) after the FOI. The potential elicited by the FOI was observed during 700 ms. Two and even three consecutive subsequent fixations could occur in this time interval; however, the overlapping effect due to this second fixation (Target+2) located at the end of the time window would be much reduced. It was even more valid for the third one. Moreover, as explained in Luck (2005), when averaging the EFRPs over several trials, the temporal jitter between two adjacent fixations acts as a low-pass filter. The EFRP responses revealed early and late potentials, which are in higher and lower frequencies, respectively. Considering the observed EFRP time locked on Target+2 (Figure 9B), we assumed that these responses do not have late potentials but rather only early potentials. After low-pass filtering (jittering effect), the result would be negligible. Similar rationales can be made for the FOIs Target-1 and Target+1. Consequently, we consider only three coupled equations (identical to Equation 1) to model the potentials elicited by the fixations before the target, in the target, and after the target. Like the ADJAR algorithm (Woldorff, 1993), an iterative procedure has been applied. In this section, we present only the predicted results on overlaps, which justify the choice of the alternative indicator. In Figure 10, for each fixation (Target-1, Target, and Target+1), the previous and subsequent potentials causing overlaps are plotted (red and blue dotted lines, respectively) at the zero onset time. The corresponding distributions of IFI values are also plotted. These overlaps were derived (bold lines). Negative IFI values, previous potentials, and overlap [OV_p(t)] are shown in red, and positive IFI values, subsequent potential, and overlap [OV_s(t)] are shown in blue. 

Figure 10

 

Illustration of the jittering effect (in red for the previous fixations; in blue for the subsequent fixations). The distributions of the IFI values are indicated by vertical bars. The components predicted by the model S(t) are indicated by the dotted line, and the jitter of these components with the corresponding distribution of IFI values is indicated by the bold line. These results are for the overlapping context on the (A) Target-1, (B) Target, and (C) Target+1 fixations for the Cz channel.

As expected the overlap OV_P(t) (Figure 10A, in bold red) on the Target-1 potential and the overlap OV_S(t) (Figure 10C , in bold blue) on the Target+1 potential were weak due to the jitter of potentials with only early potentials. For the observed EFRP at the Target-1 fixation, the overlap OV_s(t) (Figure 10A, in bold blue) due to the subsequent Target potential explained the slow-growing derive observed at the end of the time window (Figure 8B). For the observed EFRP on the Target fixation, the overlap OV_p(t) (Figure 10B, in bold red) due to the previous Target-1 potential explained the increased amplitude of the early potentials between 0 and 200 ms (Figure 6B). The amplitude stabilization of this potential at the end of the period (Figure 6B) was explained by the overlap OV_s(t) (Figure 10B, in bold blue) due to the subsequent Target+1 potential. Finally, when the waveform of the EFRP at the Target+1 fixation is observed, the amplitude enhancement of the early potentials (Figure 7B) can be explained by the overlap OV_p(t) (Figure 10C, in bold red) due to the previous Target potential. 

The lessons drawn from these results helped define a measure for evaluating and comparing the neural activities between conditions. Measures classically used, such as averaging the amplitude of the EFRPs during a specific time period after baseline correction, do not take into account overlap issues. To take into account these limitations, we proposed to do the following: 

•  
  Use a relative amplitude per condition. The origin in amplitude is set to the minimum value observed on the grand average. Let us note this amplitude a₀.
•  
  Find the start time per subject of the P300 potential (t_M). This time corresponds to the time of the minimum amplitude observed between 200 and 400 ms.
•  
  Set the stop time to end the average at 500 ms.

The model assumes that this relative measure reduced the impact of the confounding factors related to the overlaps on the evaluation of the P300 amplitude. The EEG signal was averaged between t_M and 500 ms. The two measures are defined as follows: 

•  
  The first measure, called absolute measure, is the usual average on a given interval:

 

 

 

•  
  The second measure is the proposed relative measure:

 

 

 

Statistical analyses of EFRPs

In the following, we present the two measures described previously. For both measures, for each electrode Fz, Cz, and Pz, a repeated measures analysis of variance was run on the mean amplitude of the EEG signal calculated over the defined time period with condition (i.e., FOIs) as a within-subject factor. Multiple comparisons were assessed with Bonferroni post hoc tests. 

Analysis 1

This first EFRP analysis focused on the neuronal potentials elicited by the FOIs VS target, VS salient, and FE salient. For both the absolute and the relative measure, the statistical analysis revealed that condition significantly influenced the potential amplitude for the three electrodes—absolute measure: Fz, F(2, 76) = 12.33, p < 0.001; Cz, F(2, 76) = 26.72, p < 0.001; Pz, F(2, 76) = 11.46, p < 0.001; relative measure: Fz, F(2, 76) = 15.25, p < 0.001; Cz, F(2, 76) = 29.09, p < 0.001; Pz: F(2, 76) = 18.90, p < 0.001 (Figure 11). The differences between the three FOIs are summarized in Table 5. 

Figure 11

 

Mean EEG amplitude (A) for the absolute measure and (B) for the relative measure for the FOIs VS target, VS salient, and FE salient.

Table 5

 

Summary of the significant differences for the FOIs VS target, VS salient, and FE salient for the absolute and relative measures. The dash indicates that the differences are not significant. Differences between the two measures are highlighted in bold.

 

Analysis 2

The second EFRP analysis focused on the neuronal potentials elicited by the FOIs Target, Target+1 ROI, Out, Target2, and Control. For both the absolute and the relative measure, the statistical analysis revealed that condition significantly influenced the potential amplitude for the three electrodes—absolute measure: Fz, F(4, 152) = 10.34, p < 0.001; Cz, F(4, 152) = 18.24, p < 0.001; Pz, F(4, 152) = 8.19, p < 0.001; relative measure: Fz, F(4, 152) = 5.03, p < 0.001; Cz, F(4, 152) = 12.49, p < 0.001; Pz, F(4, 152) = 14.89, p < 0.001 (Figure 12). The differences between the three FOIs are summarized in Table 6. 

Figure 12

 

Mean EEG amplitude (A) for the absolute measure and (B) for the relative measure for the FOIs Target, Target+1 ROI, Out, Target2, and Control.

Table 6

 

Summary of the significant differences for the FOIs Target, Target+1 ROI, Out, Target2, and Control for the absolute and relative measures. Dashes indicate that the differences are not significant. Differences between the two measures are highlighted in bold.

 

Analysis 3

The third EFRP analysis was divided into two parts. First, we focused on the neuronal potentials elicited by the FOIs Target-2, Target-1, Target, and Control. Then the analysis focused on the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control. 

Regarding the FOIs Target-2, Target-1, Target, and Control, for both the absolute and the relative measure, the statistical analysis revealed that condition significantly influenced the potential amplitude for the three electrodes—absolute measure: Fz, F(3, 114) = 13.36, p < 0.001; Cz, F(3, 114) = 22.83, p < 0.001; Pz, F(3, 114) = 10.49, p < 0.001; relative measure: Fz, F(3, 114) = 4.96, p < 0.003; Cz, F(3, 114) = 10.28, p < 0.001; Pz, F(3, 114) = 7.09, p < 0.001 (Figure 13). The differences between the three FOIs are summarized in Table 7. 

Figure 13

 

Mean EEG amplitude (A) for the absolute measure and (B) for the relative measure for the FOIs Target-2, Target-1, Target, and Control.

Table 7

 

Summary of the significant differences for the FOIs Target-2, Target-1, Target, and Control for the absolute and relative measures. Dashes indicate that the differences are not significant. Differences between the two measures are highlighted in bold.

 

Regarding the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control, for both the absolute and the relative measure, the statistical analysis revealed that condition significantly influenced the potential amplitude for the three electrodes—absolute measure: Fz, F(4, 152) = 10.97, p < 0.001; Cz, F(4, 152) = 20.41, p < 0.001; Pz, F(4, 152) = 9.91, p < 0.001; relative measure: Fz, F(4, 152) = 8.69, p < 0.001; Cz, F(4, 152) = 15.99, p < 0.001; Pz, F(4, 152) = 23.85, p < 0.001 (Figure 14). The differences between the three FOIs are summarized in Table 8. 

Figure 14

 

Mean EEG amplitude (A) for the absolute measure and (B) for the relative measure for the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control.

Table 8

 

Summary of the significant differences for the FOIs Target, Target+1 ROI, Target+1, Target+2, and Control for the absolute and relative measures. Differences between the two measures are highlighted in bold.

 

Discussion

The goal of this article is to study the P300 potential, well studied in ERPs, using an ecological paradigm: Observers have to search for an object embedded in natural scenes, and eye movements are necessary. Half of the scenes contained a target object, whereas the other half required observers to freely explore natural scenes without any specific task. Eye movements and EEG signals were synchronously recorded. To maintain an ecological paradigm, we did not constrain the duration of fixations, contrary to previous studies (Brouwer et al., 2013; Kaunitz et al., 2014). Fixation duration was between 200 and 300 ms, which classically is reported during the exploration of natural scenes (Castelhano, Mack, & Henderson, 2009; Tatler & Vincent, 2008). This duration is shorter than the latency of the P300 potential, which implies two consequences: (a) A saccade can occurred at the end of the time window used to study EFRPs, and (b) overlaps of early and late potentials of the previous and following fixations can modify the EFRP. To ensure that these two consequences were taken into account to explain EFRP, we followed a rigorous methodology. First, a correction of eye movements (ICA) was applied to correct artifacts due to saccades and blinks from the EEG signal. Second, for each analysis, fixations were selected depending on their duration, with the goal of equalizing the distributions of fixation durations between FOIs. Consequently, the distributions of the IFI value with the subsequent fixations were also in line across the conditions. Finally, we developed a model for expressing the observed EFRP function of the potential elicited at the onset of the target fixation but also of the potentials elicited at the onset of adjacent fixations. This model was inspired by the ADJAR algorithm and extended to three consecutive fixations. Results from this model were used to define an averaged measure on the observed EFRPs using a relative measure that reduces the impact of the cofounding factors related to potential overlaps. 

The first analysis compared the VS condition with control conditions (VS salient and FE salient). A specific eye movement pattern was observed for the VS task when the target object was present. Participants who were actively looking at the scene to find the target object performed more eye movements and had shorter fixations than during free exploration. Moreover, when the object was fixated, the amplitudes of incoming and outgoing saccades were shorter. EEG analysis replicates classical results with a P300 potential observed for the first fixation onto the target object. This potential was observed mainly on the centroparietal electrodes (Cz and Pz), showing a larger amplitude compared with control conditions (VS salient and VS target). This first result was in line with previous studies (Brouwer et al., 2013; Healy & Smeaton, 2011; Kaunitz et al., 2014) that have shown that the P300 potential was elicited over parietocentral and frontal regions. The difference was significant using both the absolute and relative measures. 

In a second analysis, we compared eye movement and neural activity for the first fixation onto the target, called target fixation, the fixation just after, and the second fixation onto the target. The fixation immediately following the target fixation (Target+1 ROI) was longer than the previous (Target), with a small saccade between. Participants made several fixations onto the same object and made small saccades between fixations, probably to confirm that they were well fixating the target object or because they had achieved their goal of finding the object. Fixations outside the target object (Out) showed an opposite pattern, with shorter duration and longer incoming and outgoing saccade amplitudes reflecting a wider exploration. For those fixations, no specific potential was elicited, confirming that the P300 potential observed for target fixations was a consequence of fixating on the target object. Additionally, a P300 potential was elicited by the second fixation onto the target. This P300 potential has the same latency but smaller amplitude than the P300 potential elicited by the first fixation to the target. However, the overlapping context was different for the first fixation onto the target and the second one. This was confirmed by the two measures used to compare potential amplitudes. With the relative measure, the P300 amplitude was higher for Target compared with Target2 for Fz, Cz, and Pz electrodes. This result had to be taken into account for the interpretation of the weaker P300 potential observed for the second target fixation, with an amplitude decrease explained by habituation as previously reported (Ravden & Polich, 1998). 

Due to the coregistration of eye movements and EEG signals, we were able to analyze the neural activity occurring before and after the first fixation onto the target object. To our knowledge, this has not previously been reported in EFRP studies. This third analysis highlighted the visual strategy adopted by participants for finding the target object. Results suggest that the scene was explored as quickly as possible until the object was found. After that, participants fixated for a longer time on the target if they stayed in the ROI before continuing the exploration. Concerning EFRPs, we analyzed late potentials for fixations occurring before and after the target fixation. The proposed model offers a means to take the overlap issue into account. Moreover, the successive FOI were compared with control fixations. 

For the fixation immediately before (Target-1), a late potential appeared but with a longer latency than the P300 potential observed for the target fixation. Using only the absolute measure, one would conclude that this potential showed smaller amplitude and might correspond to a P300 potential, linked to the detection of the target in parafoveal vision. Indeed, it has been shown that the detection of a target in parafoveal vision modified the latency of the P300 potential when participants fixated on the target (Brouwer et al., 2014), but the previous fixation was not taken into account. However, in our experiment, longer outgoing saccades (∼8°) were observed from the Target-1 fixation. Moreover, the amplitude of the observed potential was significantly lower at the onset of this fixation compared with the target fixation. This was confirmed by comparing absolute and relative measures. The model also confirmed this observation showing an increase of the amplitude directly linked to overlaps with the potential elicited at the target fixation. Taken together, we are therefore reluctant to interpret this late potential as a P300. 

Concerning fixations immediately following target fixation, they were analyzed taking into account whether the fixation following the target fixation was inside the target object (Target+1 ROI) or not (Target+1). Note that on average (in 72% of the cases), participants did two consecutive fixations onto the target object. This second consecutive fixation on the target object was a significant event with two properties: Temporally it occurred immediately after the target fixation, and spatially it was located in the target object. For this specific fixation, a large potential before and at the latency of the P300 potential was observed. The model predicted that the increase occurring early at the onset of the fixation was due to the overlap of the potential elicited at the previous fixation (i.e., the target fixation). To a lesser extent, the same reasoning could be applied to the subsequent potential (Target+2), positively biased by the overlap of the previous fixation (Target+1). These predictions were confirmed by statistical analysis using the relative measure. The potentials elicited by the target fixation as well as the following fixation (in or outside of the target object) showed a higher amplitude compared with the subsequent fixation Target+2 and our control condition. There is no difference between the amplitudes of the potentials elicited at the Target and the Target+1 ROI fixations. To solve the task, participants remained during two consecutive fixations inside the target object. For the first fixation, but also for the following fixation on the target object, the late potential elicited by this second fixation was interpreted as a P300 potential. This was consistent with an interpretation of context updating (Donchin, 1981; Polich, 2012) for which consecutive saccades were programmed to solve the visual search task. 

In this article, we proposed an ecologic paradigm associated with a rigorous methodology for studying the P300 potential and its evolution during the active exploration of natural scenes. Our approach involves four components. First, the selection of fixations based on a criterion of duration allows a fair comparison of neural activities on a late period compatible with the latency of the P300 potential. Second, the design of different analyses with a same control condition provides an anchor point for a coherent interpretation of all the results. Third, a model expressing the elicited potentials at three consecutive fixations in relation to their overlaps provides guidelines for the interpretation of the observed EFRPs. Last, following the prediction of the model, a relative scale is preferred to the usual absolute scale for averaging the neural amplitude to compare. These two last points are the main contributions of this article and a first step in helping the interpretation of EFRPs in natural contexts. To our knowledge, there is no algorithm for the computation of evoked potentials, efficiently benchmarked in the context of overlapped EFRP estimates. By using this original methodology, a P300 potential elicited by the fixation on the target object for the first time was reported. This potential was also elicited at the subsequent fixation remaining on the target object. 

Finally, in line with classical ERP results and EFRPs studies in visual search, these results were reported in a less constrained experimental procedure. The interest of this study came from the study of EFRPs continuously in time on adjacent fixations during an ecological paradigm. A model had been designed to help the interpretation by the separation of cofounding factors between the potentials of interest and overlaps without controlling eye movements. The next step is to use this model not only for the interpretation of the EFRPs but also directly for these potential estimates. 

Acknowledgments

This work was supported by a grant from the ANR Gaze-EEG “Joint synchronous EEG signal and eye tracking processing for spatio-temporal analysis and modeling of neural activities” and a grant from the French Ministère de la Recherche et de l'Enseignement Supérieur funding the doctorate work of Hélène Queste-Devillez. A part of the software development was performed by Ronald Phlypo, Nicolas Tarrin, and Gelu Ionescu. We are grateful to Holly Earls for her thorough language check. 

Commercial relationships: none. 

Corresponding author: Anne Guérin-Dugué. 

Email: [e-mail]. 

Address: Université Grenoble Alpes, GIPSA-Lab, Grenoble, France. 

Footnotes

¹  www.grenoblecognition.fr.

Footnotes

²  www.robots.ox.ac.uk/∼vgg/data/oxbuildings.

Footnotes

³  Data analyzed in this study are part of a larger experiment with four different visual tasks: free exploration, scene categorization, visual search, and spatial organization.

Footnotes

⁴  Median values were used because of the skewed distributions of fixation durations and saccade amplitudes.

Appendix A: Denoising ocular artifacts by ICA

Before the ERPs analysis, preprocessing was applied. Epochs for which EEG signals were noisy or contained artifacts due to muscular activity (Supplementary Figure S1) were suppressed after visual inspection (6.82 ± 1.01 epochs on average per participant and per condition; Supplementary Table S1, column 1). In our experiment, an epoch, starting 500 ms before the scene onset and lasting 4 s (duration of the scene presentation), corresponds to a trial. 

Supplementary Figure S1.

 

Two examples of EEG epochs suppressed after visual inspection: EEG signal for the 28 electrodes (positioned according to the extended 10-20 system) and eye movement data for four channels (eyeLx, eyeLy, x and y coordinates of the left eye, and eyeRx and eyeRy for the right eye), plus one binary channel to indicate blinks (blink). Black curves represent the raw signal; red curves represent the noisy part of the signal. (A) All electrodes are noisy due to muscular artifacts, and (B) one electrode is noisy due to nonphysiological noise.

Supplementary Table S1.

 

Statistics (M and SD) for the preprocessing steps for the FE and VS tasks.

 

To identify and eliminate the components of EEG signals corresponding to artifacts due to eye movements (saccades and/or blinks), an ICA (infomax ICA, EEGlab) was conducted, which was preceded by a PCA (retaining number of channels minus one component—i.e., 27). Both the PCA and the ICA were applied for all the epochs per participant and per condition (39 × three separated analysis). ICA is commonly used in ERP and EFRP studies to remove eye movement artifacts in EEG signals (Frey, et al., 2013; Graupner et al., 2007; Nikolaev et al., 2011; Ossandón et al., 2010). Classically, the ICA correction is executed automatically on the basis of reference signals such as vertical and horizontal electro-oculograms (EOG). In our study, the selection of independent components related to eye movements was not automatic and was conducted according to the procedure described below. 

The identification of the independent components related to eye movements (saccades and/or blinks) was based on (a) the temporal evolution of the components relative to the EEG and eye tracking signals (Supplementary Figure S2, bottom; step executed for all epochs) and (b) the topographic map and spectral activity of the components (Supplementary Figure S3; topography and spectral activity were identified by averaging the component's signal over all trials). 

Supplementary Figure S2.

 

Top: EEG signals for two epochs for 11 frontal electrodes. Pink indicates the part of the signal that corresponds to (A) saccades and (B) blinks. Middle: Eye tracking signals for four channels (eyeLx, eyeLy, x and y coordinates of the left eye, and eyeRx and eyeRy for the right eye) and one binary channel to indicate blinks (blink). Bottom: A selection of nine independent components (over the 27 components resulting from the ICA; components are displayed in decreasing order of the EEG variance accounted for by each component). Orange indicates the preselected independent components.

Supplementary Figure S3.

 

The maps for two preselected independent components (IC8 and IC2) corresponding to (A) saccades and (B) blinks. Bottom: The EEG spectrum. Top left: The scalp map. Top right: The component activity.

First, based on visualization of the independent components and EEG and eye tracking signals for each epoch, we preselected the independent components whose evolution coincided with those of blinks and saccades (Supplementary Figure S2). In a second step, the preselected independent components were validated on the basis of their spectral activities and topographic maps (Supplementary Figure S3). The smoothly decreasing EEG spectrum is typical of an eye artifact. For topographic maps, artifacts related to saccades and blinks are characterized by a strong opposed left and right projection (saccades) and a strong far-frontal projection (blinks). 

Following these two steps, on average, 3.54 ± 0.27 components per participant and per condition were selected and removed from the EEG signal (Supplementary Table S1, column 2). After the ICA, the EEG signal was reconstructed (Supplementary Figure S4) and used for further ERP and EFRP analyses. 

Supplementary Figure S4.

 

The EEG signal before (in black) and after (in blue) the ICA correction for (A) saccades and (B) blinks for 11 frontal electrodes.

A second visual inspection was then performed after the ICA correction to suppress epochs for which EEG signals were not effectively corrected (Supplementary Figure S5; Supplementary Table S1, column 3). However, we know that ICA correction is not always sufficient and that artifacts linked to eye movements may still persist. Furthermore, although the ICA correction was used with success in previous studies, it has been shown that ICA can distort cerebral sources besides attenuating ocular signals (Keren, Yuval-Greenberg, & Deouell, 2010). 

Supplementary Figure S5.

 

Top: The EEG signal before (in black) and after (in blue) the ICA correction for an epoch rejected after ICA correction for a bad correction of a blink. Bottom: Eye tracking signals for four channels (eyeLx, eyeLy, x and y coordinates of the left eye, and eyeRx and eyeRy for the right eye) and one binary channel to indicate blinks (blink).

Supplementary Table S1 summarizes for each visual task (VS and FE) the mean number of segments removed in each preprocessing step (columns 1 and 3) and the average number of independent components removed by our ICA procedure (column 2). The mean number of remaining segments used for EFRP analyses is given in the last column. 

Appendix B: Eye movement data after fixations selection for EFRP analysis

The selection procedure for the fixations was applied in order to adjust the same empirical distribution of fixation durations before EFRP analysis. In this Appendix, the statistics on eye movement data are revisited for the three analyses. For each analysis, we present the result of the selection procedure by showing the empirical distribution of fixation durations before and after selection (Supplementary Figures S6, S8, S10, S12). For EFRP estimates, the IFI values between adjacent fixations are important criteria. By a selection based on the fixation durations, we expected a matching of the three conditions on the distributions of IFI values with the subsequent fixations. However, the distribution of IFI values with the previous fixations was not directly affected by the selection procedure, so significant differences could be observed after the selection. Finally, the saccadic context was revisited after the selection procedure (Supplementary Figures S7, S9, S11, S13). One table per condition summarizes the statistics on IFI and saccadic context after the selection procedure (Supplementary Tables S2, S3, S4, S5). 

First fixations in the ROIs (I)

Supplementary Figure S6.

 

Distributions of fixation durations of the first fixation in the ROIs for the conditions VS target, VS salient, and FE salient (A) before and (B) after matching the distributions of fixation durations. On the remaining fixations, there were no significant differences between distributions (Chi-square test, df = 16, p = 0.903), whereas differences were significant before selection (df = 18, p < 0.001).

Supplementary Figure S7.

 

Distributions of orientations for the saccade before (incoming) and after (outgoing) the first fixation in the ROIs for the conditions VS target, VS salient, and FE salient.

Supplementary Table S2.

 

Statistics (Chi-square test) on the distribution of IFI and saccadic parameters after the selection procedure for the conditions VS target, VS salient, and FE salient.

 

Fixations onto the target object and after (II)

Supplementary Figure S8.

 

Distributions of fixation durations of the fixations onto the target object and after for the conditions Target, Target+1 ROI, Out, Target2, and Control (A) before and (B) after matching the distributions of fixation durations. On the remaining fixations, there were no significant differences between distributions (Chi-square test, df = 32, p = 0.373), whereas differences were significant before selection (df = 48, p < 0.001).

Supplementary Figure S9.

 

Distributions of orientations for the saccade before (incoming) and after (outgoing) the fixations onto the target object and after for the conditions Target, Target+1 ROI, Out, Target2, and Control.

Supplementary Table S3.

 

Statistics (Chi-square test) on the distribution of IFI and saccadic parameters after the selection procedure for the conditions Target, Target+1 ROI, Out, Target2, and Control.

 

Fixations onto the target object and previous (IIIA)

Supplementary Figure S10.

 

Distributions of fixation durations of the fixations before and onto the target object for the conditions Target-2, Target-1, Target, and Control (A) before and (B) after matching the distributions of fixation durations. On the remaining fixations, there were no significant differences between distributions (Chi-square test, df = 21, p = 0.533), whereas differences were significant before selection (df = 27, p < 0.001).

Supplementary Figure S11.

 

Distributions of orientations for saccade before (incoming) and after (outgoing) the fixations before, onto the target object, and after for the conditions Target-2, Target-1, Target, and Control.

Supplementary Table S4.

 

Statistics (Chi-square test) on the distribution of IFI and saccadic parameters after the selection procedure for the conditions Target-2, Target-1, Target, and Control.

 

Fixations onto the target object and subsequent (IIIB)

Supplementary Figure S12.

 

Distributions of fixation durations of the fixations onto the target object and subsequent for the conditions Target, Target+1, Out, Target 2, and Control (A) before and (B) after matching distributions the fixation durations. On the remaining fixations, there were no significant differences between distributions (Chi-square test, df = 50, p = 0.086), whereas differences were significant before selection (df = 52, p < 0.001).

Supplementary Figure S13.

 

Distributions of orientations for saccade before (incoming) and after (outgoing) the fixations before, onto the target object, and after for the conditions Target, Target+1 ROI, Target+1, Target+2, and Control.

Supplementary Table S5.

 

Statistics (Chi-square test) on the distribution of IFI and saccadic parameters after the selection procedure for the conditions Target, Target+1 ROI, Target+1, Target+2, and Control.

 

Appendix C: EFRPs for all electrodes

Supplementary Figure S14.

 

EFRPs for the 28 electrodes and the three experimental conditions (I) VS target, VS salient, and FE salient.

Supplementary Figure S15.

 

EFRPs for the 28 electrodes and the five experimental conditions (II) Target, Target+1 ROI, Out, Target2, and Control.

Supplementary Figure S16.

 

EFRPs for the 28 electrodes and the four experimental conditions (IIIA) Target-2, Target-1, Target, and Control.

Supplementary Figure S17.

 

EFRPs for the 28 electrodes and the five experimental conditions (IIIB) Target, Target+1 ROI, Target+1, Target+2, and Control.