Distinguishing between target and nontarget fixations in a visual search task using fixation-related potentials

Abstract

Abstract

Abstract:

Abstract  The P300 event-related potential (ERP) can be used to infer whether an observer is looking at a target or not. Common practice in P300 experiments and applications is that observers are asked to fixate their eyes while stimuli are presented. We investigated the possibility to differentiate between single target and nontarget fixations in a target search task involving eye movements by using EEG epochs synchronized to fixation onset (fixation-related potentials: FRPs). Participants systematically scanned search displays consisting of six small Landolt Cs in search of Cs with a particular orientation. After each search display, they indicated whether and where target Cs had been presented. As expected, an FRP component consistent with the P300 reliably distinguished between target and nontarget fixations. It was possible to classify single FRPs into target and nontarget FRPs above chance (on average 62% correct, where 50% would be chance). These results are the first step to practical applications such as covertly monitoring observers' interests and supporting search tasks.

Introduction

The P300 is an event related potential (ERP) occurring approximately 250–500 ms after a target or task-relevant stimulus has been presented (Ravden & Polich, 1999). Because the P300 is relatively easy to detect and its amplitude depends on voluntarily controlled endogenous attentional processes, it is often used as a control signal in brain–computer interfaces (BCIs; Brouwer & van Erp, 2010; Farwell & Donchin, 1988; Jin et al., 2012; Sellers, Krusienski, McFarland, Vaughan, & Wolpaw, 2006). In P300 BCIs, different stimuli are presented sequentially. The stimulus chosen by the observer (i.e., the stimulus that the observer focuses his/her attention on), elicits a P300 that is detected by the computer. In this way, BCI users can select one of several presented options, such as a particular letter to spell a word. 

Usually, participants using a P300 BCI or taking part in an experiment in which the P300 is investigated, are asked not to move their eyes around the time that the P300 occurs. For example, they fixate a fixation cross that is subsequently replaced by a particular visual stimulus, or they fixate a target among nontargets and count the number of times it is flashed. However, in natural visual search tasks, observers sample their visual environment by self-initiated fixations and saccades instead of fixating a location where the visual stimulus is known to appear. We expect that the brain's electrophysiological response to perceiving a target among nontargets will be similar regardless of whether the eyes are static and targets and nontargets are presented at a fixation location or whether an observer fixates a set of nontargets individually in search for a target (Kamienkowski, Ison, Quiroga, & Sigman, 2012). Here we try to infer from EEG whether individuals look at a target object or not in situations with natural eye movements. Rather than locking EEG to stimulus onset, we lock EEG to fixation onset and examine whether we can distinguish target from nontarget fixations on a single-fixation basis. If so, this would enable new types of (online) applications, such as covertly monitoring observers' interests and supporting search tasks, as well as further stimulating more ecologically valid scenarios in EEG studies about visual search, selective attention, and detection. 

Fixation-related potentials (FRPs) or saccade related potentials (SRPs) have been studied in the context of reading (Baccino & Manunta, 2005; Dimigen, Sommer, Hohlfeld, Jacobs, & Kliegl, 2011; Marton, Szirtes, & Breuer, 1985; Simola, Holmqvist, & Lindgren, 2009), viewing and identifying drawings (Ravden & Polich, 1999), viewing natural images (Ossandón, Helo, Montefusco-Siegmund, & Maldonado, 2010), studying awareness of oculomotor errors (Belopolsky, Kramer, & Theeuwes, 2008) and, as an example of an applied setting, evaluation of lighting systems (Yagi, Imanishi, Konishi, Akashi, & Kanaya, 1998). It is generally held that conventional ERP components such as the P1 and the N1 can be identified in FRPs or SRPs (Baccino & Manunta, 2005; Belopolsky et al., 2008; Kazai & Yagi, 2003; Ossandón et al., 2010; Rämä & Baccino, 2010) as well as later components such as the N400 (Dimigen et al., 2011). A review of studies on FRPs and SRPs in reading is given by Dimigen et al. (2011). They conclude that while EEG is seldom recorded in natural viewing conditions, it can indeed contribute new answers to long-standing questions in the field of reading. 

We are interested in an FRP component related to the P300 that could distinguish between target fixations and nontarget fixations. In research by Hale, Fuchs, and Berka (2008), observers viewed photographic images of industrial sites and satellite images in which they had to search for specific targets, such as a particular vehicle. An overall difference between different types of target and nontarget FRPs was found. In particular, figure 2 in Hale et al. (2008) shows a late, broad, positive peak at around 600 ms for FRPs associated with fixations on correctly identified targets (hits) compared to other FRPs (such as those associated with correct rejections and misses). While the grand average results are clear, it is not clear whether or not one individual FRP could be labeled as belonging to a target or nontarget fixation. In addition, the difference between target and nontarget FRPs that they found may have been due to several confounding factors. First, saccades to targets may have systematically differed from saccades to nontargets (for instance, saccades to targets may have been shorter than those to nontargets), which could have led to different effects of eye movements on target FRPs than on nontarget FRPs (Plöchl, Ossandón, & König, 2012). Second, the difference between target and nontarget FRPs could have been caused by the effect of preparing to push a button to indicate that a target was found. The results may also have been affected by systematic differences in search times of images with and without a target present since, in case of target present images, a search ended when a target was found. Finally, low-level visual differences between targets and nontargets may have contributed to their results. It has been shown that properties of the visual stimuli such as luminance and spatial frequency can indeed affect SRPs (Marton & Szirtes, 1982; Ossandón et al., 2010; Yagi, Ishida, & Katayama, 1992). While in practical applications confounding effects like the ones just mentioned could occur and may even be used, we wanted to investigate in the present experiment to what extent target and non-target FRPs can be distinguished. 

A poster by Luo, Parra, and Sajda (2009) describes a classification approach where SRPs locked to single saccades toward targets are distinguished from single nontarget SRPs. Both EEG epochs before and after saccade onset could be classified above chance as being associated with targets or nontargets. While classification results were good, these may be explained by factors other than brain signals associated with top-down attentional processes. In the study by Luo et al. (2009), observers were presented with several image chips scattered across a screen. Targets were chips containing people; nontarget chips did not contain people. Thus, and as also discussed and indicated by their results, low-level (bottom-up) target saliency effects could have contributed to increased classification accuracy. There could be one or no target present on the screen and participants pressed a button after identifying a target or after deciding that no target was present. This means that target fixations were associated with button presses and nontarget fixations occurred on average earlier in a trial than target fixations, which may also have contributed to the difference between target and nontarget SRPs. Luo et al. (2009) report timing and scalp topographies to not reflect typical P300 results (where timing, at least, would indeed be expected to be atypical since in their design, objects could be identified to be targets or not before fixation). 

A recent study (Kamienkowski et al., 2012) compared target and nontarget FRPs while taking care of the confounding effects as mentioned above. They asked their observers to freely search a group of stimuli, consisting of 20 instances of the letter “E.” Two of the Es were mirror images and constituted the targets. Each E was surrounded by nine symbols (“#”) such that participants were required to fixate the letters for identification. In this way, it was guaranteed that target detection could only occur after target fixation onset and not before. Participants pressed a button after they found the second target, which terminated the trial. Only target FRPs associated with the first target were included in the analysis. Nontarget FRPs were selected in such a way that they matched the target FRPs in terms of accompanying saccade length and direction. Kamienkowski et al. (2012) found a P300-like late effect at around 480 ms at Cz and Pz. They also found a difference between target and nontarget FRPs at Cz and Pz after about 150 ms. This effect was not present in data from fixed gaze control experiments. Kamienkowski et al. (2012) conclude that this early effect is consistent with presaccadic attentional engagement enhancing rapid processing of target identification. Kamienkowski et al. (2012) did not present analyses to classify single trials. 

Similar to Kamienkowski et al. (2012) we aimed to design our experiment in such a way that eye movements and other confounding effects as discussed above cannot explain potential differences between target and nontarget FRPs. As in Kamienkowski et al. (2012), we used stimuli that required foveal vision in order to be identified as a target or nontarget, and that did not differ on low-level visual features, such as luminance and spatial frequency. Rather than selecting (nontarget) data afterward in order to compare target and nontarget FRPs unconfounded by saccade length and direction, we asked participants to search a series of circularly arranged, equally distanced objects following a predetermined scan path. Targets could be present on any location and participants did not know the number of targets present beforehand. They indicated whether and where targets were present after finishing the complete fixation sequence. 

Methods

Participants

Thirteen participants (four female and nine male, between 21 and 34 years old) were recruited through the participant pool of the Netherlands Organization for Applied Scientific Research (TNO). They received a monetary reward to make up for their travel and time. One additional participant (36 years old, female, participant 1 in Table 1) was the first author. The study is in accordance with the Declaration of Helsinki and has been approved by the local ethics committee. All participants signed an informed consent form prior to taking part in the experiment. 

Table 1

 

Number of valid FRPs per participant, classification performance as expressed by mean Cohen's kappa K and the standard deviation of K.

Participant	Number of included FRPs	Classification performance (K)	Standard deviation of K
1	358	0.37	0.12
2	315	0.21	0.08
3	162	0.21	0.13
4	305	0.31	0.10
5	79	0.44	0.18
6	148	0.28	0.03
7	152	0.17	0.10
8	191	0.20	0.12
9	82	−0.02	0.24
10	253	0.23	0.10
11	437	0.18	0.12

Apparatus

Stimuli were presented on a 17-inch flat-screen monitor (Dell 1707FP), set at a resolution of 1280 × 1024 pixels. The refresh rate for this screen was set at 60 Hz. 

Eye position and blinks were recorded at 50 Hz using a Tobii x50 eye tracker (Tobii Technology, Stockholm, Sweden). This system consists of a noninvasive stand-alone unit positioned underneath the stimulus screen. 

EEG was recorded at Fz, Cz, Pz, Oz, P3, P4, PO7, and PO8 electrode sites of the 10-20 system using electrodes mounted in an EEG cap (G.Tec Medical Engineering GmbH, Schiedlberg, Austria). The EEG electrodes were referenced to linked mastoid electrodes. EOG electrodes were fitted to the outer canthi of both eyes, as well as above and below the left eye (Kendall Neonatal ECG electrodes, Tyco Healthcare Deutschland GmbH, Neustadt, Germany). The horizontal EOG electrodes were referenced to each other and the vertical EOG electrodes likewise. The impedances of all electrodes were below 5kΩ. EEG and EOG data were sampled at 256 samples per second, and were filtered by a 0.1 Hz high pass, a 100 Hz low pass, and a 50 Hz notch filter using a USB Biosignal Amplifier (G.Tec Medical Engineering). 

Stimuli

Figure 1B gives an impression of the search display. It consisted of six Landolt Cs with four possible orientations: the gap could be at the top, bottom, left, or right. The Cs were arranged in a circle with a diameter of 960 pixels (23.82 degrees of visual angle) and displayed at the 12, 2, 4, 6, 8, and 10 o'clock positions. They were 15 pixels (0.38 deg) in diameter with a gap size of 3 pixels (0.08°). The shortest distance between two Cs was 6.29°, making it impossible to detect the orientation of any C other than the one currently fixated. The target C could have any orientation, but remained the same for each individual participant. The nontarget Cs had randomly selected other orientations. One-third of the displays contained two targets, one third contained one target, and one third contained no targets so that participants could (almost) never know whether the next C would be a target. Target positions were randomly selected and could be adjacent, but targets were never presented at the 12 o'clock position. The C at 12 o'clock was only used as a starting and ending point of the eye movement sequence since FRPs associated with fixations on the 12 o'clock position could differ from those on other positions because of anticipatory eye movements. 

Figure 1

 

Overview of the course of a trial: (A) presentation of the target, (B) search display, and (C) display to indicate target positions. For clarity, Landolt Cs in Panel B are represented substantially larger with respect to the circle than they actually were. The arrows and words in Panel B clarify the scan path and were not present in the actual display.

Task, design, and procedure

Participants seated themselves comfortably in front of the screen. They were instructed to minimize eye blinks, and head and body movements. This was further accomplished by the use of a chinrest. The distance between the screen and the eyes was 60 cm. A nine-point calibration was used to calibrate the Tobii eye tracker. Figure 1 gives an overview of a trial. A trial started with the presentation of the target in the center of the screen (Figure 1A), both as a reminder of what the target orientation was, and as a fixation point for the participant. After a mouse click by the participant on the “next” button, the search display appeared (Figure 1B). Participants were instructed to fixate on each C, starting at the C at the 12 o'clock position and switching to the next in clockwise direction as fast as they wished. When the participants returned to the top C, they indicated this by pressing the “next” button. Upon clicking, the Cs in the search display were replaced by buttons (Figure 1C). Participants were asked to click any button corresponding with a previously displayed target C. When finished, they clicked “next” and a new trial started. Each participant was tested in four blocks, each consisting of 60 trials. 

Analysis

Tobii data and default ClearView 2.7.1 algorithms (Tobii Technology) were used to identify fixations made throughout the experiment and inspect their locations. The ClearView algorithms require subsequent valid data points to reside within an area of 60 pixels (1.52°) for a minimum of 100 ms in order to be included in a fixation. These requirements imply exclusion of blinks (closed eyes prevent recording of valid data points). Fixation location is the average of those data points. We then selected fixations on the Cs at the 2, 4, 6, 8, and 10 o'clock positions. Participants were considered to fixate a C when the fixation was within an area of 100 × 100 pixels (2.52°) centered on the middle of the C. Subsequently, EOG was used to define fixation onset that was exactly synchronized with EEG. For this, we looked for the peak EOG speed within a window starting 160 ms before and ending 160 ms after fixation onset as defined by the Tobii. The start of the fixation was then set at the first frame that the speed was below 2 mV/frame. 

FRPs were defined as EEG samples starting at fixation onset and ending 500 ms after, where only the first fixation on a particular C was included in the analysis. The interval of 500 ms is a compromise between including the window of the expected effect (which is rather late) and not losing too much data: since in this study we are interested in clean data reflecting the P300, FRPs associated with fixations shorter than 500 ms were discarded (see also Kamienkowski et al., 2012, who used a similar selection criterion for a similar purpose). For three participants, this resulted in less than 10 valid target fixations. These participants were therefore excluded from analysis. For the remaining participants, the 500 ms requirement decreased the total number of valid fixations from 9552 to 2624. FRPs associated with misses (target Cs that were not identified as such) were discarded; this happened only in 17 of the 2624 fixations. The fixations did not include false alarms (nontarget Cs that were indicated to be targets). Thus, in the remainder, target FRPs are always associated with hits and nontarget FRPs with correct rejections. Finally, we discarded FRPs containing absolute voltages exceeding 80 mV (126 FRPs), which left us with 2481 fixations. Table 1 indicates the number of FRPs that were used in the analysis for each participant. 

While the probability of target presentation in the experiment was equal for each of the 2, 4, 6, 8, and 10 o'clock positions, the selection of data may have affected this. As a check, we determined the proportion of nontarget and target fixations included in the analysis for each participant and each location. The amount of included fixations appeared to depend on object location (with 28%, 19%, 6%, 9%, and 39% for the 2, 4, 6, 8, and 10 o'clock positions, respectively). However, paired t tests indicate that there is no difference between proportion target and nontarget fixations for the different locations, except for the location at 6 o'clock (8% target fixations vs. 3% nontarget fixations, t₁₀ = 2.78, p = 0.02). 

Since we are interested in judging from a single FRP whether or not a specific observer is looking at a target, we used an analysis based on machine learning methods. For each participant, we estimated whether FRPs (running from fixation onset until 500 ms later) could be correctly classified as associated with a target or nontarget. These estimates were produced using a five-fold cross-validation procedure in which data was partitioned into five subsets. A classification pipeline was trained on four of the subsets, and evaluated on the remaining test subset. This procedure was then repeated for all subsets, resulting in five performance measures per participant. 

The classification pipeline consisted of the following steps. First, we computed a symmetrical spatial whitening transform P to normalize the channel covariance matrix S:  

 

This transformation has the property that it attenuates strong signals, and amplifies smaller signals. The FRPs were subsequently spatially filtered by P. The resulting 8 (electrodes) by 129 (samples) arrays were rearranged to form 1,032 dimensional feature vectors. An L2-regularized logistic regression (LR) classifier was trained on these feature vectors. We chose logistic regression since it provides probabilistic output, and works robustly on imbalanced datasets. For each fold, the whitening transformation and the coefficients of the LR classifier were re-estimated using the training set. The LR classifier has a hyperparameter, C, that controls the amount of regularization to reduce the chance of overfitting. The value of C was optimized using a second five-fold cross-validation procedure performed on the training data of the outer fold. This guarantees that all adjustable parameters were optimized independently of the test sets. This classification pipeline was implemented using the version 0.13.1 of the scikit-learn machine learning package (Pedregosa et al., 2011). 

If the FRPs contain no information that distinguishes between targets and nontargets, the classifier's best guess would be whichever of the two classes occurs most often. The percentage of the most-frequently occurring class thus represents chance level. Because the balance between target and nontarget FRPs included in the analysis differs between participants, the expected percentage of correct classifications, P(e), for a random classifier also differs between participants. Therefore, we use Cohen's kappa, K, to describe classification performance. It linearly transforms this percentage such that 0 corresponds to chance level agreement, and 1 corresponds to perfect agreement between the random predictions and the true labels of the FRPs:  

 

where P(a) is the mean correspondence of the classifier and the labels, and P(e) is expected chance level correspondence. 

For practical applications, it would be convenient to classify fixations with durations shorter than 500 ms. To assess the feasibility of using these shorter windows, and to examine information content over time after fixation, we repeated the analysis presented above on the same data but for eight shorter window sizes. The start of the window was kept at the onset of the fixation. Tested window lengths were 441, 379, 316, 254, 191, 129, 66, and 8 ms. Note that decreasing the window length reduces the dimensionality of the feature space, which could improve performance, while on the other hand, valuable information may no longer be contained in the analysis window, which could decrease performance. 

We compare the classification scores obtained using EEG with scores of the exact same classification analysis performed on EOG, and on EOG combined with EEG recordings. This was done to verify that potentially distinguishing information in EEG did not originate from eye movements. Pairs of performance measured by Cohen's kappa were compared using Wilcoxon signed-rank tests. 

As another comparison for the EEG classification, we determined how well target and nontarget fixations can be distinguished on the basis of fixation duration. For this analysis, we did not exclude fixations on the basis of fixation duration, which made it possible to use data for all subjects. For each participant, a subset of nontarget fixations was chosen to have equal-sized sets of target and nontarget fixations. Then, the median fixation duration was determined per participant. Fixations with a duration longer than the median were classified as target fixations; shorter fixations were classified as nontarget fixations. If there were fixations with a duration equal to the median, half were classified as target and half as nontarget fixations. 

For each participant and electrode, we also calculated grand average target and nontarget FRPs. These served as input for calculating standard errors of the mean per frame, as well as for running paired sample t tests that were used to test for significant differences between target and nontarget FRPs (alpha of 0.01). In addition, we checked for significant differences in the 100 ms epoch before fixation onset to verify that FRPs do not differ before fixation onset. Note that the conventional procedure of using ERP baselines (subtracting the mean amplitude of the ERP over an interval before stimulus onset) is not appropriate here because at the interval before fixation onset the eyes are moving, which results in a strongly variable signal. This means that our FRPs are expected to be more variable than conventional baseline corrected ERPs. To verify that EOG associated with target and nontargets is similar, as intended by our experimental design, we examined EOG “FRPs” associated with the same set of fixations as used in the FRP analyses in the same way. 

Results

The solid lines in Figure 2 show the grand average target and nontarget FRPs (with equal weights given to participants) at each of the EEG recording sites. The averages are consistent with a higher P300 for a target than for a nontarget fixation. The difference is significant at some frames towards the end at the parietal and parieto-occipital electrode sites. At POz a clear P1 is visible. Except for one frame at PO8, the 100 ms epoch before fixation onset does not show differences between targets and nontargets, which is consistent with identification of the C happening only after fixation onset. The clear positive parietal peak corresponds to a presaccadic spike potential (Balaban & Weinstein, 1985; Carl, Açık, König, Engel, & Hipp, 2012; Dimigen et al., 2011). The dotted lines in Figure 2 show the standard error of the traces averaged over subjects for each of the EEG electrodes. Variability is largest around fixation and strongly decreases going from frontal to more occipital electrodes, suggesting that eye movements contributed to the variability of the signal. t tests on the EOG “FRPs” (Figure 3) and the 100 ms before did not indicate significant differences between targets and nontargets, confirming that saccades to targets and nontargets were similar. 

Figure 2

 

Average target (bold solid lines) and nontarget (thin solid lines) FRPs and the 100 ms before for each of the electrode locations. Blue crosses indicate significant differences between the two as indicated by running paired sample t tests. The dashed lines indicate standard errors of the mean of target (bold) and nontarget (thin) FRPs.

Figure 3

 

Average target (bold solid lines) and nontarget (thin solid lines) EOG “FRPs” and the 100 ms before. Running paired sample t tests did not indicate significant differences. The dashed lines indicate standard errors of the mean of target (bold) and nontarget (thin) EOG “FRPs.”

Classification performance per participant, as measured by Cohen's kappa, K, is presented in Table 1. With the exception of participant 9, all participants have a classification performance well above chance level. This indicates that the EEG can be used to predict whether a user is fixating on a target or not. The mean K over participants is 0.23 (SD 0.11), which is significantly above chance level of 0 as indicated by a Wilcoxon signed-rank test (t₁₀ = 1, p < 0.01). With equal numbers of targets and nontargets, this K corresponds with an accuracy of 62% correct predictions. With 39% target and 61% nontarget FRPs (the average proportions of included FRPs in our classification analysis), this K corresponds with an accuracy of 70% correct predictions. Since the five parts of the cross-validation procedure are not independent, we cannot simply check significance on the individual participant level. There appears to be a correlation between the amount of training data and the classification performance, which is significant with Spearman's rank correlation (r = 0.61, p = 0.047). This suggests that better results can be obtained when more FRPs are available for training the model. 

Figure 4 shows the influence of the length of the FRP window on classification performance. Performance increases with the length of the window. There is a plateau with relatively constant performance for windows from fixation onset up to 250 ms later. After 300 ms, performance steadily increases with longer windows. This is consistent with the P300 contributing to the distinction between target and nontarget fixations. 

Figure 4

 

Classification performance (K) for differently sized time windows, all starting at fixation onset. Dashed lines indicate standard errors of the mean.

Comparing K resulting from the analysis on FRPs as described above to K resulting from the same analysis but based on EOG, indicates that classification performance is genuinely based on EEG. Although the EOG channels perform slightly above chance with a mean K of 0.13, EEG significantly outperforms EOG as indicated by a Wilcoxon signed-rank test (t₁₀ = 7, p = 0.02). In addition, when we compare K resulting from the analysis on EEG to the same analysis performed on a combination of EEG and EOG, we find no significant difference (t₁₀ = 32, p = 0.93). 

As expected, fixations on targets generally last longer than fixations on nontargets: median target fixation durations averaged over participants was 415 ms (SD 75); for nontargets this was 371 ms (SD 93). A paired t test indicated that this difference was significant (t₁₄ = 3.95, p < 0.01). Classification analysis based on fixation duration resulted in an average classification accuracy of 56% (range 46%–66%, SD 6%). Excluding the three participants that were left out in the EEG analysis did not change these numbers, except for an increase of the standard deviation to 7%. 

Discussion

In the current study we provided evidence for a target-linked P300 in a well-controlled search task involving eye movements, where ERPs were locked to fixation onset rather than to the onset of an externally imposed stimulus. To the best of our knowledge, Kamienkowski and others (2012) is the only well-controlled study that has demonstrated this before. Additionally, we showed that individual FRPs can be labeled offline as belonging to a target or nontarget fixation significantly above chance. Note that classification analysis as used here is based on response patterns of individual observers, while these patterns may not be visible in overall FRP averages. 

Our interest here were brain responses to self-paced fixation of top-down determined targets versus nontargets. We included only long fixations in our analyses in order to examine late FRP components that were not contaminated by eye movements. We do have to note that our eye movement recording equipment did not allow checking for microsaccades (Martinez-Conde, Otero-Millan, & Macknik, 2013), which may have affected FRPs through microsaccade related brain activity (Dimigen, Valsecchi, Sommer, & Kliegl, 2009; Yuval-Greenberg, Tomer, Keren, Nelken, & Deouell, 2008). EOG does not seem to differ significantly between targets and nontargets. More importantly, classification based on EOG performs significantly worse than classification based on EEG, while adding EOG to EEG classification models does not improve performance. This is consistent with brain signals related to target identification underlying the distinction between targets and nontargets. 

Consistent with Dimigen et al. (2011) and Kamienkowski et al. (2012), our findings emphasize that, for later components as well, it is possible to investigate ERPs in contexts with self-paced eye movements and suggest that P300 knowledge that has been gathered within tasks with stationary eyes could be generalized to situations with eye movements. We hope this will spur more ecologically valid visual P300 studies in which participants can explore and sample a visual environment themselves rather than being presented with specific stimuli at specific times. Furthermore, these findings are of relevance in applied fields of research such as passive Brain-Computer Interfaces (Coffey, Brouwer, Wilschut, & van Erp, 2010; Zander & Kothe, 2011; Zander, Kothe, Welke, & Roetting, 2008), augmented cognition (van Erp, Veltman, & Grootjen, 2010; Schmorrow, Estabrooke, & Grootjen, 2009) and neuroergonomics (Parasuraman & Wilson, 2008). The aim of these fields is to exploit spontaneously occurring brain signals (and possibly other physiological signals or types of information) in order to smooth human–machine interaction or to support users in other ways. While augmented cognition and passive BCI focus on online use of brain signals, offline use of brain signals that reflect the state of the user could also be useful when, for instance, evaluating different interfaces or studying task performance over time. 

Using FRPs online or offline would make it possible to infer whether, in situations with natural eye movements, individuals look at target objects (objects that are of specific relevance to the observer) or not, without the need to ask them to report. FRPs would be especially valuable in situations where you do not want conscious judgments to interfere with what is being reported as a “target,” for example, when evaluating advertisements. Another group of relevant cases is that which involves the observer being more or less unaware of the target. FRPs could be used to extract implicit knowledge from expert observers or would enable the support of search tasks (e.g., by asking radiologists looking for tumors “didn't you just miss something interesting?”). In this last type of use it is not strictly necessary to know exactly what a user was looking at—EEG and EOG sensors would suffice to indicate when something interesting was fixated. Another important issue is whether FRPs can distinguish between correct rejections and misses (i.e., fixations on nontargets and targets in cases when an observer does not report a target), and between hits and false alarms (i.e., distinguish between targets and nontargets when the observer reports a target). If these were possible, FRPs would provide information that is impossible to obtain by asking observers to report. Several ERP studies suggest that subliminal oddball stimuli indeed elicit P300s (Bernat, Shevrin, & Snodgrass, 2001; Brazdil, Rektor, Dufek, Jurak, & Daniel, 1998; Devrim, Demiralp, & Kurt, 1997). Recently, Zander, Gaertner, Kothe, and Vilimek (2011) showed that selection systems based on eye gaze duration could be improved by adding consciously generated brain signals (power in certain EEG frequency ranges). Our findings suggest that spontaneously generated ERP features could also be of help. However, while the single-fixation classification results show that it is possible to distinguish between target and nontarget fixations based on fixation-locked EEG above chance, an average performance of 62% (if chance is 50%) may not be good enough for many practical applications. Selection of long fixations will also limit practical use. Possible routes for improvement are discussed below. 

One way to improve classification is simply to acquire more data to train the classifier. We did not reach ceiling level with respect to the amount of training data as indicated by our finding that participants who produced more training data generally reached higher classification performance. Furthermore, the probabilistic output of the classifier can be exploited to better interpret data or improve the accuracy of detections. For example, when the FRP classifier yields uncertain predictions, other information can be relied on more heavily. In any case, other information besides EEG will probably increase to classification performance. Here we showed that fixation duration contains information. Factors that we carefully tried to exclude from affecting our results in the current study might be exploited in real life to reach better distinction between targets and nontargets. For instance, saccade length may be informative where saccade length might be used directly in the estimate, or indirectly through affecting EEG differently for targets than for nontargets. Also, context information and low-level visual features may be used (e.g., certain fixations that are largely determined by bottom-up processes may be predicted a priori). Finally, we would like to note that FRPs shorter than 500 ms also contain information as showed by our classification analysis of differently sized intervals. This is consistent with Kamienkowski et al. (2012) who found a relatively early component (around 150 ms) distinguishing between targets and nontargets. They point out that in contrast to (experimental) situations that impose stimuli to stationary eyes, the brain may be sensitized for information uptake in a case of a self-paced fixation (presaccadic attentional engagement). Also, in other search tasks, targets are detected in the periphery, that is, before the target is fixated, causing a positive target potential to occur close to fixation onset, as we demonstrated in a later experiment (Brinkhuis & Brouwer, 2012). 

In conclusion, our findings of a P300 like component in target fixation-locked ERPs and the fact that target and nontarget FRPs can be distinguished on a single trial level could be of importance both in fundamental and applied fields of ERP research on visual search and attention. 

Acknowledgments

The authors thank Malte Risto for his work on a pilot version of the experiment, Bart Kappé for brainstorming about the design of the current experiment and Pjotr van Amerongen for programming the experiment. The authors gratefully acknowledge the support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science. 

Commercial relationships: none. 

Corresponding author: Anne-Marie Brouwer. 

Email: [e-mail]. 

Address: TNO Perceptual and Cognitive Systems, Soesterberg, The Netherlands.