Comparing eye movements to detected vs. undetected target stimuli in an Identity Search task

Abstract

Why do we perceive some elements in a visual scene, while others remain undetected? To learn about the sequence of events leading to detection, we directly compared fixations on detected vs. undetected items. Our novel Identity Search task display comprised twelve cards, all different except for two pairs of identical cards. Participants search for one pair. Task properties allow us to monitor fixations on distinct card regions and study search dynamics. We find that detected pair cards were fixated more often and for longer times than undetected pair cards. Within the search sequence, there are fewer intervening fixations between detected than undetected pair cards. Only at an advanced stage of the search do fixations on pair cards become closer. We suggest that both the absolute number of fixations and their temporal proximity influence detection. In the dynamics of search, a bifurcation point is observed, when these differential characteristics begin. Analysis of the break point in the sequence of fixations on to-be-detected cards suggests that there is an early—perhaps unconscious—recognition stage, followed by more fixations and only later by detection. We suggest that several target fixations are needed for processing visual information to achieve recognition.

Introduction

When viewing a natural scene, there are some elements that we consciously perceive, while others escape our notice. That we do not perceive all the details in a scene has been eminently demonstrated by the phenomena of change blindness (Rensink, O'Regan, & Clark, 1997; Simons & Levin, 1997), repetition blindness (Kanwisher, 1987, 1991), and the attentional blink (Potter, Chun, Banks, & Muckenhoupt, 1998; Raymond, Shapiro, & Arnell, 1992; Shapiro, Raymond, & Arnell, 1994). A possible source for this conscious/non-conscious perception dichotomy has been detailed in Reverse Hierarchy Theory (Hochstein & Ahissar, 2002; see also Ahissar & Hochstein, 1997, 2004). Some elements in a scene attract what has been called exogenous attention (Jonides, 1981; Müller & Rabbitt, 1989; Posner, 1980), leading to such phenomena as feature search pop-out (Treisman & Gelade, 1980). Attention may be overt (with the eyes fixating the attended element) or covert (without such eye movements). Similarly, even with attention, detection may be conscious and reportable or unconscious, leading to positive and negative priming effects without observer awareness (DeSchepper & Treisman, 1996; Treisman, 2006). 

It is well known that eye movements reflect cognitive processes (Henderson & Hollingworth, 1999; Liversedge & Findlay, 2000; Rayner, 1998; Ringach, Hawken, & Shapley, 1996; Stone, Miles, & Banks, 2003). Fixation locates a particular part of the visual scene on the fovea, and this input, including approximately a radius of 2° of visual field (Anstis, 1974; Riggs, 1965), undergoes more major processing. Thus, the sequence of fixations, which is controlled by high cortical levels (Bruce, Goldberg, Bushnell, & Stanton, 1985; Chen & Zelinsky, 2006; Schall, 1991), also affects the subsequent high-level processing that will occur in our brains. 

Nevertheless, much remains unknown concerning the dynamics of eye movements and the causal relationship between eye fixations and perceptual cognition. How do fixations influence perception and how does perception influence fixations? 

Classical studies concluded that more informative scene regions receive more fixations (Antes, 1974; Buswell, 1935; Loftus & Mackworth, 1978; Mackworth & Morandi, 1967; Yarbus, 1967), but does the visual system know which regions are informative before boosting fixations upon them? In contrast, our (non-natural scene) displays contained two target pairs, which were equally informative. We do not compare local visual features of objects or the semantics of the objects in the scene. Rather, we examine how two stimuli compete for perceptual and cognitive processing, and how fixations influence—and especially how they are influenced by—this processing. 

In contrast to change blindness, which deals with detection over time, we devised a task that deals with search and detection over space. We ask what determines which elements will be foveated. In particular, when observers perform a search task requiring conscious perception and comparison among a number of target elements, what will be their sequence of saccades and fixations on the different elements within the scene? Will there be a phase of concentrated fixations on the targets that will ultimately be found—even before they have been (consciously) found? When there is more than one target in a scene, what determines which target is found and which remains unknown? Does conscious target detection come before or after concentrated target fixation? What role do fixations play in the search process? In particular, what is the relationship between repeated fixations on the same scene region and limited Working Memory capacity? What in the sequence of fixations reflects or influences ultimate conscious perception? 

We tracked eye movements to study the sequence of perceptual events leading to conscious recognition and detection. Similar studies have been performed with the change-blindness paradigm (Droll, Gigone, & Hayhoe, 2007), investigating which changes in a scene are perceived and what is the sequence of eye movements that precede change detection. As mentioned, unlike the change-detection task, in our study, comparisons are made across space rather than over time. Furthermore, we include two target pairs in each task display and study perceptual effects by comparing fixations on detected vs. undetected stimuli, in the same display. For this purpose, we use a novel task that we introduce here—the Identity Search task. 

The Identity Search task display contains computer screen “cards”, each with a square array of scrambled black and white square units. The subjects' task was to find two exactly identical cards. Try to find such a pair in Figure 1. You may have noticed that there are two such pairs in this example. This is not by chance—it is exactly the point. The displays are specially designed to suit our purpose of differentiating between detected and undetected target pairs. Within each display, all cards are different, except for exactly two pairs of identical cards. The subjects' task was to find one identical pair in each display—not being informed that two target pairs were present. Thus, in all cases one pair is detected and one is not, allowing us to compare properties of the detected and the undetected pairs and the pattern of eye movements to each. 

Figure 1

 

The Identity Search task. A number of “cards” are shown to the subject, whose task is to find two identical cards. For our experiments, we used the version shown, including 12 cards, each marked with a 4 × 4 scrambled array of black and white squares. Can you find two identical cards? Actually, there are two such pairs in the demonstrated figure, as there were in all displays in the experiment. (Here, card numbers 4 and 9 and card numbers 3 and 8 are each an identical pair; card numbering starts from the top left and goes row by row.)

When a pair is found and marked, the entire display is replaced for the following trial. The number of cards, array size, and number of black square units on each card are parameters under experimenter control but, in our experiments, were always set at 12 cards, 16 units per card (in a 4 × 4 array), and half of the units on each card were black, and half were white. 

The Identity Search task includes the following characteristics that are important for the goals of our study: 

1.  
   Each display includes one detected and one undetected target pair for comparison.
2.  
   Displays are divided into distinct search and eye-fixation regions (the different cards).
3.  
   Identical card pairs do not pop out. Rather, the pre-recognition process requires several seconds of search (allowing us to follow the dynamics of eye movement patterns during the course of the search process).
4.  
   The search process culminates in a momentary (though not immediate) recognition (an insight or “aha” experience; see Ahissar & Hochstein, 1997; Rubin, Nakayama, & Shapley, 1997), abruptly ending the search process.
5.  
   An enormous number of novel displays may be created, allowing us to repeat the task with a new search and a new “aha” experience each time. Elaborating, there are  

   $\left(\begin{array}{c}16\\ 8\end{array}\right)=12,870,$

   (1)

   combinations for each card (denoted as x), therefore  

   $\prod _{i=0}^{11}(x-i)\approx 2·{10}^{49},$

   (2)

   combinations of different displays.
6.  
   The task is easy to master, so that learning is rapid (requiring less than an hour of practice with the task), and performance stabilizes to a steady level.
7.  
   Task complexity may be controlled (by varying the number of cards and the size of the array on each), to reach a desired average detection time. (For the present experiment, this determination was done in a preliminary pilot experiment and we report here results with a constant set of parameters.)

We previously studied performance on a computerized version of the Set game (Set Enterprises Inc.; Jacob & Hochstein, 2008). This game also has many of the characteristics listed above. The Identity Search task preserves the benefits of the Set game and has the advantage that it is less complex and more easily learned. This new task was devised based on the finding that Set search depends on similarity detection. 

We examine fixations on the detected pair compared to those on the undetected pair in the same display. Regarding the number of fixations, one could expect one of two effects of cards belonging to an ultimately detected pair: either there will be fewer fixations on the detected pair, or more fixations on the detected pair (see also Nodine, Carmody, & Kundel, 1978). For each effect, several scenarios can explain the result. For example, if detection is a result of an inherent property of some of the pairs, then those pairs would be detected immediately upon being observed, and the answer will be fewer fixations on the ultimately detected pair cards. If, on the other hand, detection of a target requires several stages until culminating in explicit recognition, then repeated observations might take place in earlier stages of the perceptual process, and the answer will be more fixations on the ultimately detected pair cards. Our current study aims at differentiating between these possible scenarios. 

Methods

Display images

We implemented the Identity Search task with a Matlab GUIDE (Graphical User Interface Design Environment). Each display had 12 cards (3 rows by 4 columns), each represented as a click-able button, with an image of a scrambled 4 × 4 square array of black and white square units on each card. Half (8) of these squares were black and half were white. The background between the cards was gray (RGB: 153, 153, 153). Within each display, all cards are different, except for exactly two pairs of identical cards (each pair is unique and different from the other pair), located randomly. Each card occupied 2.7° × 2.7° of visual field. The space between the cards was 2.7° vertically and 2.9° horizontally. 

The task

Subjects were instructed to mark (by mouse click) two identical cards. They were asked to mark these cards as quickly as possible but to try to avoid mistakes. They were not informed that two such pairs were present. During each trial, if a player changed his or her mind after marking one card, they could “unmark” the card by a second mouse click on it. Upon correct designation of a pair, a “continue” button appeared if it was a valid identical pair; otherwise a message “try again” appeared on the top of the screen (and the Response Time clock continued running until correct identification). Upon pressing the “continue” button, the entire display was replaced for the following trial. Response Time (RT) was measured from pressing the “continue” button until choosing the second card in the pair; subjects were informed of this timing procedure. 

Each subject performed the task first without eye-movement tracking for a training session of 100 trials (with a pseudorandom distribution of 1–5 card pairs). Then their performance was measured with eye tracking in a single session of 50 trials (with 2 pairs per display; each trial lasted 8.3 s, on average). 

Participants

Eight university students participated in the full course of 100 training trials followed by 50 trials with tracked eye movements (two subjects had also participated in other variations of the game, as a part of a pilot study). Thus, we gathered for our analysis a total of 400 trials with eye movements. Subjects were remunerated for participation. Sessions lasted about an hour, including eye-tracker adjustment, calibration, and validation. In only 3 of the 400 trials, subjects marked 2 wrong cards and were informed that they do not form a pair; in another 6 trials, they marked one card and then “unmarked” it. 

Equipment

We used the Eyelink eye tracker (SR Research Ltd., Ontario, Canada), based on two infrared light-emitting diodes (IR-LEDs) in front of each eye and a 250-Hz camera that records the LED reflections of the corneas. Similarly, head-movement compensation is done by the same principle: 4 IR-LEDs at the display monitor corners are detected by a cyclopean camera. Pupil position is sampled once every 4 ms, for each eye. Subjects sat 80 cm from a Samsung SyncMaster 19″ (18″ viewable) CRT monitor, with 4:3 format, and 800 × 600 pixel resolution. Thus, the foveal visual field of 2° occupies 2.8 cm or 67 monitor pixels. The monitor was surrounded by a black screen. 

Calibration was done using the built-in 9-point calibration grid and was followed by the validation. We corrected for equipment drift by verifying gaze points every 10 trials, using this verification to correct preceding and following trials. 

The EyeLink was run with Matlab Psychophysics and Eyelink Toolboxes (Brainard, 1997; Cornelissen, Peters, & Palmer, 2002). Analysis of fixations and saccades was done with the EyeLink program. 

Analysis

The relevant data for this study were the eye fixations. Subjects mark the first pair that they find, resulting in a detected target pair and one that is left undetected. We compare fixations on this detected pair and those on the undetected pair. Of course there were also many fixations on the other 8 cards, but these are all “dead-ends” that cannot lead to pair detection. 

We recorded from both eyes, but the analysis was performed according to the dominant eye, which has priority in visual processing (Shneor & Hochstein, 2006). The dominant eye was determined by the Porta Test (a sighting test in which observers position a near stimulus, such as a finger, so that it appears collinear with a distant stimulus; Porta, 1953). 

For eye-position analysis, the display was divided into 12 equal regions, each including one of the cards and its surrounding region (up to the border halfway between it and the adjacent card or the screen edge). Card regions are 176 × 170 pixels (∼5.7° × 5.5° of visual field). For each pair (detected and undetected), we analyzed the total number of fixations (on the 2 card regions of the pair), the total viewing time of the two card regions, and the sequence of fixations—the sequential distance between fixations on the 2 card regions of the pair. We excluded from these and all following analyses the period during which subjects marked the cards, so that this period does not bias the results, i.e., all fixations ending after the first click were excluded. Note that those fixations are shown in Figure 2 in order to provide a complete picture, but they were not included in the analyses in any way. We studied the means of each parameter as reflecting global results over all trials and also analyzed temporal variations of the parameters to determine search dynamics within the trial. 

Figure 2

 

Two examples of eye-movement records during search for an identical pair of cards. Identical card pairs are indicated by different colors: the eventually detected pair in red, the undetected pair in blue. (a) In this example, there were 7 fixations on the detected pair and 3 on the undetected, out of a total of 31 fixations (see Figure 6; excluding from the count the final period when the subject marked the cards, which was excluded from the analyses and appears here in a different color). The sequential number of each fixation is indicated in the circle surrounding the fixation center. (b) Five fixations on detected pair; 2 fixations on undetected pair; a total of 17 fixations (see Figure 7). The duration of each fixation is indicated near the fixation point.

Trials with exceptionally long RTs (8) were discarded (setting a bound at mean RT + 2 times the global SD). We also disregarded (7) trials with too few successfully registered fixations, which could have resulted from blinking by the subject or incorrect equipment readings (e.g., fixations beyond the monitor range) perhaps due to extreme subject changes of head or body postures; 385 trials remained. From these, we excluded trials without any fixations on the undetected pair, leaving 294 trials for full analysis. When two successive fixations were to the same card region, we combined them and counted them as a single fixation with duration equal to the sum of their durations. While there may be good reasons for not making the above two data analysis choices, we took this route since it is the more conservative; the trends we find would be enhanced by making the opposite choices. 

Results

Mean Response Time (RT) for pair detection and mouse click marking for the 294 trials was 8.7 ± 0.3 s (mean ± SE); range 2.2–24.7 s; median 7.3 s. 

Figure 2 demonstrates eye-movement records of two different trials. They are typical in that, in each case, there are fixations on various cards including on the ultimately detected pair of identical cards (framed in red) and the alternative target pair that was not detected (framed in blue), followed by the subject marking a pair of target cards (fixations during the marking period are shown in green). What can we learn about the detection process from the eye-movement records? 

Total number of fixations on pair cards

We compare the total number of fixations on the 2 pair cards, for detected vs. undetected pairs, using only the 294 trials where there was at least one fixation also on an undetected pair card. The overall number of fixations on the detected pair cards was greater than that on the undetected pair cards. Average results for all 294 trials with at least one fixation on both pairs (combining successive fixations to the same regions) were 4.6 ( SE: 0.16) fixations on the detected pair cards and 3.2 ( SE: 0.13) on the undetected pair cards. A two-way ANOVA with detected/undetected and subject as main factors showed significance for both: detected/undetected: F = 53.9, p < 0.001; subjects (as random factor): F = 4.66, p < 0.05; interaction effect insignificant: F = 0.8, p = 0.59. Thus, there were significantly more fixations on the detected pair cards than on the undetected pair cards, and this preference is consistent across subjects. 

Comparing the number of fixations on a trial-by-trial basis also showed that there were generally more fixations on the detected pair cards than on the undetected pairs for each subject, and for the entire range of 50 trials for each subject, as follows: Figure 3a shows a scatter plot by subject comparing average number of fixations on the detected pair and on the undetected pair. The points for every subject fall above the diagonal line of equality. (The regression line anchored to the origin is y = 1.44 x; 95% confidence intervals for the slope are 1.33–1.55.) Figure 3b shows that the same is true on a trial-by-trial basis for all subjects ( y = 1.21 x, C.I. 1.15–1.28). (Note that the number of fixations cannot be zero since we include only trials with at least one fixation on each pair.) 

Figure 3

 

Analysis of the number of fixations on detected vs. undetected pair cards. The top row (a, b, c) presents data for the entire trials of each subject, while the second and third rows (d, e, f and g, h, i) present data separately for the first and second halves of each trial, respectively. (a, d, g) Scatter plots by subject comparing average number of fixations on the detected pair and on the undetected pair. The points for every subject fall above the diagonal of equality, both for the total data (a) and especially for the second half of each trial (g). (b, e, h) Color-scaled plots comparing trial-by-trial (294 trials) average number of fixations on the detected and undetected pairs. Again, most points lie above the line of equality for the total data and especially for the second half of each trial. (c, f, i) Normalized difference between numbers of fixations on the two pairs on a trial-by-trial basis. Red dots represent trials of more fixations on the detected pair and blue dots represent trials of more fixations on the undetected pair. Red–blue colors are used consistently also in following figures for detected and undetected pairs, respectively. Note that there are more red dots than blue dots. Furthermore, the red dots are farther from the zero line of equality. The first half of each search does not show a difference between detected and undetected pairs. The effect of more fixations on detected pair cards is seen in the second half of each search.

There were more trials (68.4%) with a greater number of fixations on the detected pair cards than trials with a greater number of fixations on the undetected cards (15%), as shown in Figure 3c. Here we plot the normalized difference between the numbers of fixations on the two pairs, i.e., the difference between number of fixations on the detected and undetected pairs, divided by their sum. Red dots represent trials of more fixations on the detected pair and blue dots represent trials of more fixations on the undetected pair. There are more red dots than blue dots and the red dots are further from the zero line of equality (average normalized difference +0.34 vs. −0.24). As shown in the figure, in 68% of the trials there were more fixations on the detected pair cards; in 15% of the trials there were more fixations on the undetected pair cards; and in the remaining (17%) there were equal numbers of fixations on the two pairs. 

We conclude that between the two possibilities of fewer or more fixations on the to-be-detected cards, we find that there were significantly more fixations on the detected cards. 

Total viewing time of pair cards

In addition to the above comparison regarding the number of fixations per pair, we also compared the total viewing time (that is, the total duration of all fixations) on the detected and undetected pair cards. The rationale for performing this comparison in addition to the previous one is that fixation durations vary and may independently show a preference for the detected or undetected pair cards. 

Average results for all 8 subjects and 294 trials were total viewing time of 1,361 ms ( SE: 50) on the detected pair cards and 831 ms ( SE: 38) on the undetected pair. A two-way ANOVA with detected/undetected and subject as main factors showed significance for both: detected/undetected: F = 44.3, p < 0.001; subjects: F = 3.86, p < 0.05; interaction effect insignificant: F = 1.76, p = 0.09. Thus, there was significantly longer total viewing time on the detected pair cards than on the undetected pair cards, and this preference is consistent across subjects. 

Comparing the total viewing time on a trial-by-trial basis also showed that there were generally more fixations on the detected pair cards than on the undetected pairs for each subject and for the entire range of 50 trials for each subject. Figure 4a shows a scatter plot by subject comparing average total viewing time on the detected pair and on the undetected pair. The points for every subject fall above the diagonal of equality. (The origin-anchored regression line is y = 1.66 x; 95% C.I. 1.47–1.85.) Figure 4b shows that the same is true on a trial-by-trial basis for all subjects ( y = 1.3 x; C.I. 1.21–1.39). 

Figure 4

 

Analysis of total viewing time on detected vs. undetected pair cards. Data are presented in the same format as in Figure 3, with columns presenting scatter plots by subject comparing average total viewing times on the detected pair vs. on the undetected pair (left; a, d, g); scatter plots comparing trial-by-trial total viewing times on the detected and undetected pairs (middle; b, e, h); and the normalized difference between total viewing time on the two pairs, on a trial-by-trial basis (right; c, f, i). The top, middle, and bottom rows present data for the entire trials of each subject, for the first and second halves of each trial, respectively. The points for every subject fall above the diagonal of equality, both for the total data and especially for the second half of each session. The effect of longer viewing time on detected pair cards is seen in the second half of each search.

Figure 4c shows for each trial the normalized difference between total viewing time on detected and undetected pair cards; there are many more trials of longer total viewing time on detected pair cards than on undetected pair cards (more red than blue dots; 82.65% vs. 17%) and the red dots are further from the zero line of equality (average normalized difference +0.36 vs. −0.19). 

Thus, total viewing time was also consistently longer for the detected pair. 

Average fixation duration

The average duration per fixation on detected pair cards was 305 ms ( SE: 5.6), and the average duration per fixation on undetected pair cards was 260 ms ( SE: 5.3; over the 294 trials with fixations on both the detected and undetected pairs, combining successive fixations to the same regions and accumulating their duration). A two-way ANOVA with detected/undetected and subject as main factors showed significance for both: detected/undetected: F = 23.02, p < 0.005; subjects: F = 5.28, p < 0.05; interaction effect insignificant: F = 1.67, p = 0.11, confirming that this difference was consistent across subjects. We may therefore conclude that the difference in total viewing time reflects not only the difference in number of fixations but also a difference in individual fixation durations. 

Search sequence (intervals between fixations on pair cards)

Does the proximity of fixations on the two cards of the pair influence pair detection and/or is it influenced by (prior) perception? We looked at the sequence of fixations and analyzed the sequential distance (i.e., number of intervening fixation steps between fixations on the cards of the pair). An illustration of part of a search sequence, showing successive fixations to the different card regions, is presented schematically in Figure 5a. Note that after we combined successive fixations to the same card region, a sequential distance of 1 must reflect fixations on 2 different pair cards. This is not the case for fixations that are 2 or more steps apart, which can be on the same card region. We ask if sequential distance is a differentiating factor between detected and undetected pairs. 

Figure 5

 

(a) Schematic illustration of a sequence of successive fixations on different card regions within a trial. In this example, card numbers 3 and 7 form a pair and fixations on their regions within this sequence are marked in red. (b) Average occurrences per trial (over 208 trials with at least 2 fixations on each pair) of each sequential distance between fixations on detected (red) and undetected (blue) pair cards. Note that there are more occurrences with fewer intervening fixations for the detected pair cards, i.e., the red plot is shifted upward for the lower sequential-distance values compared to the blue plot. Arrows indicate mean sequential distance between fixations on the same pair.

We measured the occurrences of each sequential distance, both for detected and undetected pair cards over all 208 trials (with at least 2 fixations on each pair, so that at least one interval is obtained). Division by the number of trials yields the results shown in Figure 5b, the average number of times that each sequential distance appeared per trial. Surprisingly, none of the averages reaches even 1, i.e., no sequential distance appeared in every trial. Note that summing the averages for each pair type (detected or undetected) gives the average number of fixations on that type, minus 1, because we are counting the intervals between fixations. 

The results show more occurrences with smaller sequential distances for the detected pair cards. That is, even though there are generally more fixations on the detected pair, all the probabilities do not rise equally; rather, there is an increase only for smaller sequential distances. This difference is also expressed by the means of the distributions, indicated by arrows in Figure 5b: a sequential distance of 4.1 between fixations on detected pair cards and a sequential distance of 4.8 for undetected pair cards. These are significantly different: t-test over trials: p < 0.001. This result suggests that detected pair cards are not only viewed more often, and for longer durations, but they are also viewed in closer proximity, at least in parts of the search. We relate in the Discussion section to the interdependence of these measures. 

Division of each search trial into two halves

We examined whether the effects of more and longer fixations on detected pairs are constant over different search stages. For this purpose, we divided each search trial into its first and second halves and repeated the above analyses of total number of fixations and total viewing time for each half trial. Results are shown in the second and third rows of Figures 3 and 4. 

The first half of the search, over all 294 trials, does not show a difference between detected and undetected pairs in the number of fixations. Average results were 1.59 ( SE: 0.09) fixations on the detected pair cards and 1.77 ( SE: 0.08) on the undetected pair cards. 

Figure 3d shows a scatter plot by subject comparing average number of fixations on the detected pair and on the undetected pair. The points for every subject are gathered around the diagonal of equality, and even fall a little below it ( y = 0.91 x; C.I. 0.84–0.98). The image in Figure 3e shows that the same is true on a trial-by-trial basis for all subjects ( y = 0.91 x; C.I. 0.85–0.98). Note that here (as opposed to Figure 3b) the number of fixations can be 0, because there need not be a fixation on each pair in this half of the trial. Figure 3f shows the normalized difference between the numbers of fixations on the two pairs. Overall, considering only the first half of each trial, in 28.9% of the trials there were more fixations on the detected pair cards; in 39.8% there were more fixations on the undetected pair cards; in 26.5% there were equal numbers of fixations on the detected and undetected pair cards; in the remaining 4.8% there were no fixations on either of the pairs (and they were fixated only in the second half of the trial). 

In contrast to this result, there is a large effect on the number of fixations in the second half of each trial: Average results were 2.98 ( SE: 0.09) fixations on the detected pair cards vs. 1.45 ( SE: 0.08) on the undetected pair cards. Thus, in Figure 3g, the points for every subject fall above the diagonal of equality ( y = 2.07 x, C.I. 1.82–2.33). The image in Figure 3h shows that the same is true on a trial-by-trial basis for all subjects ( y = 1.32 x, C.I. 1.23–1.42). Overall, in 77.2% of the trials there were more fixations on the detected pair cards; in 8.5% there were more fixations on the undetected pair cards; and in 14.3% there were equal numbers of fixations on the detected and undetected pair cards, as shown in Figure 3i. 

The same difference between the two halves of the trial is seen for the total viewing time measure. The first half of the search, over all trials, does not show a difference between detected and undetected pairs. Average results were total viewing time of 410 ms ( SE: 25) on the detected and 472 ms ( SE: 24) on the undetected pair cards. 

The points in the scatter plots of Figures 4d and 4e are gathered around the diagonal of equality and even fall a little below it ( y = 0.88 x; C.I. 0.81–0.96, and y = 0.7 x; C.I. 0.62–0.77, respectively). Figure 4f shows the normalized difference between the numbers of fixations on the two pairs. Overall, longer viewing times were found in 40.1% of the trials for the detected and in 54.4% for the undetected pair cards; in 0.7% there were equal viewing times on the two pairs and in the remaining 4.8% there were no fixations on either. 

In the second half of each trial, on the other hand, large differences were found. Average total viewing times were 950 ms ( SE: 32) on the detected and 359 ms ( SE: 21) on the undetected pair cards. In Figures 4g and 4h, the points for every subject fall above the diagonal of equality ( y = 2.66 x; C.I. 2.23–3.1, and y = 1.57 x, C.I. 1.4–1.74, respectively). Overall, in 89.1% of the trials there was a longer viewing time on the detected pair cards; in 10.9% there was a longer viewing on the undetected pair cards (average normalized difference +0.6 vs. −0.27). 

The fact that there is such a difference between the first and second halves of the trial—though trial length varies over a large range—is a first strong indication that there is a correlation—and we will suggest a causal dependence—between fixation number (and duration) and detection of the identical pair. 

Search dynamics

Number and sequence of fixations

Encouraged by the results of the division into halves of each trial, we followed the dynamics of search through each trial. Results for two sample trials are shown in Figures 6 and 7. In Figures 6a and 7a, we plot the accumulated number of fixations on pair cards as a function of the successive fixations within the trial. Upward slopes reflect a fixation on a pair card and the lines remain horizontal for fixations on non-pair cards. Note that there are similar numbers of fixations on the to-be-detected (red) and the undetected (blue) pairs—up to a point where the number on the detected pair increases above the level for the undetected pair cards. 

Figure 6

 

Search Dynamics. (a) Accumulated number of fixations on detected (red) and undetected (blue) pair cards for one trial (shown in Figure 2a) as a function of the successive search fixations. Each increment corresponds to a fixation on a pair card. Fixations on the other 8 card regions leave both the pairs without an increment. The vertical line toward the end of the trial represents the point where the first card was marked (fixations after this line were not included in analyses and results). (b) Dynamics of fraction of fixations on pair cards, as a function of number of search fixations. (c) Fraction of fixations, averaged over a running boxcar with a width of 6 fixations. (d) Sequential distance from the previous pair card, for each fixation on a pair card. The first fixation for each pair is arbitrarily plotted at zero. Numbers inside circles represent the card region on which the fixation fell.

Figure 7

 

Search Dynamics. Another example, as in Figure 6, corresponding to the example shown in Figure 2b.

In Figures 6d and 7d, we plot the sequential distance between each two fixations on the pair cards (as appears in Figure 5). An opposite effect is seen from the number of fixations on the pair cards: As the search progresses, the number of intervening fixations between the detected pair card fixations becomes smaller, i.e., there is a gradually decreasing sequential distance between fixations on the to-be-detected pair cards. 

Fraction and running boxcar average

To further quantify the search dynamics, we analyzed two measures of the fixations on the pair cards. As in the examples of Figures 6b and 7b, we studied the fraction of fixations on the pair cards, for each successive fixation number. We also ran boxcar averaging over the number of fixations on the detected and undetected cards, and then calculated the fraction for this box size. Examples are shown in Figures 6c and 7c, using a 6-fixation boxcar width. Note that using a running average may more closely reflect the influence of memory for the most recent fixations. 

In both analyses, a clear point is seen where the curves for the to-be-detected and the undetected pair cards diverge, and fixations on the detected pair rise above those on the undetected pair. This divergence reflects a change in the relative fraction of fixations on the two pairs during the course of the search, indicating that a specific event has taken place in the detection process. 

Bifurcation point

To further investigate the issue of dynamics, we wish to systematically study the point where a bifurcation occurs between fixations on the detected and the undetected pairs in the search dynamics—the point where the ultimately detected pair “overpowers” the other potential target. Such an observation can be seen for single trials in Figures 6a, b, c and 7a, b, c, for the accumulated number of fixations on the detected pair in comparison to the undetected pair—or its running boxcar average—during progress of the search process. Since different trials have different search times and a wide range of numbers of fixations, we cannot simply average over trials of different subjects or even of a single subject. 

To average over trials and subjects, we measured the backward dynamics—the accumulated number of fixations on pair cards (as in Figures 6a and 7a) but now relative to the actual detection point (i.e., we look at fixations on the target pairs as a function of the number of fixations before the first mouse click on a card). 

Results are shown in Figure 8a. We plot the accumulated fixations on the detected and undetected pair cards, averaging over different trials and subjects, aligning the results according to the end of the trial, the time of marking detected cards. Shown on the same graph are the results for two trial-length ranges: short trials of 8–19 fixations and longer trials with 20–80 fixations. Figure 8b plots the slopes of the accumulated number of fixation graphs of Figure 8a. Again, the different windows are superimposed on the same graph. 

Figure 8

 

Backward Dynamics. (a) Average over all trials of the backward dynamics—the accumulated number of fixations on pair cards (as in Figures 6 and 7a) aligned to the actual detection point (i.e., the first mouse click on a card); dashed line: trials with 8–19 fixations; solid line: trials with 20–80 fixations; red: detected pairs; blue: undetected pairs. Curves for the two ranges show a very good fit, besides an upward shift for the range of longer trials (see text). Error bars indicate SE between trials; arrow indicates where the slope (b) changes significantly—a change that is regarded as the bifurcation point. (b) Slope of the accumulated number of fixations (as in a). The slope indicates the probability that the next fixation will be on a pair card. Error bars indicate 95% C.I. There is a clear bifurcation point where the slope of the detected pair rises above that of the undetected one and its slope exceeds a pre-determined threshold. The time of this bifurcation in (b) is indicated by an arrow in (a).

Methodological note: In order to perform a backward dynamics analysis for a particular number of fixations before initial card marking, there have to be at least that number of fixations in the trial. Thus, very short trials will be excluded from this analysis. For this reason, to compare short and long trials, we applied two different “window” sizes: a window of 8 preceding fixations (trials in the range of 8–19 fixations in total), and a window of 20 preceding fixations (trials with 20–80 fixations in total). 

Three characteristics are immediately apparent: (1) The patterns of the fixations on the detected and undetected pairs are nearly identical up to a point approximately 4 fixations before the end of the trial. At this point, there is a marked change in the pattern for the detected pair, with the number of fixations showing a sharp upturn ( Figures 8a and 8b). This is seen most clearly as a point where the slope of the detected pair rises above the slope of the undetected pair ( Figure 8b). (2) This divergence between the detected and undetected pairs occurs a while before initiation of the card-marking process, ∼5 fixations or about 1.5 s before the first mouse click ( Figure 8b). (3) The point of divergence in slopes is only slightly shifted to the right for the short search sequences. Surprisingly, these characteristics are the same for short and long trials, i.e., there is no dependence on history. We conclude that something important occurs at this point. 

To elaborate, the two different time windows show a very good fit, except for an upward shift for the longer trials. This shift is due to a side effect of using a longer window: the total number of fixations increases, therefore slightly shifting up the number of fixations on the pairs. Yet, the average increases by only about 3 fixations on the detected pair, in the whole range between trials with 8–19 fixations and trials with 20–80 fixations. This relatively small increase implies that no matter how many fixations there are in total—the number of fixations on pair cards needed for identification is more or less fixed (within a certain range); see Figure 9. 

Figure 9

 

(a) Histogram of number of fixations for each trial on the detected pair cards, over all trials. (b) Psychometric curve of percent detection as a function of the number of fixations on the detected pair cards.

Discussion

We compared fixations on detected and undetected target items and found that there are more fixations on the ultimately detected pair cards than on the undetected ones. This is reflected in the proportion of trials in which there are more fixations on detected pairs (68.4% vs. 15%) and the average number of fixations on detected pair cards (4.6 vs. 3.2). The increase in number of fixations is confirmed by an increase in total viewing time on detected pair cards, and we found that the average duration of each fixation was also longer for the detected pair. 

In addition, the results also show fewer intervening fixations on other cards between fixations on the detected pair cards in the search sequence. The average sequential distance between fixations on card regions is smaller for the detected than for the undetected pairs. 

In fact, if there are more fixations on the detected pair cards, then automatically the average interval or sequential distance between them will be smaller—these are two sides of the same coin. Still, it is of interest to examine both, since we do not know, a priori, which one affects perception—the number of fixations on the cards (together with their duration, affecting total viewing time) or the temporal adjacency of the fixations. Further study is required to differentiate between these interdependent parameters, e.g., by varying the number of cards in the display, which can separately affect the number of fixations on the detected pair or the sequential distance between such fixations. As discussed below, we believe that these two effects work together to bring about conscious perception of the target pair. 

As mentioned in the Methods section, we combined successive saccades that fell on the same card region. Which choice is the correct one? If the two-saccade sequence is pre-planned, it may be correct to consider them as two separate fixations on the card, rather than as a single fixation with a “correction” in the middle. Recent studies of saccades to long or short words (Vergilino & Beauvillain, 2001) or to elongated or separated shorter objects that were displaced during the first saccade (Vergilino-Perez & Findlay, 2006) suggest that successive saccades to the same object may be very different than saccades to different objects, perhaps reflecting pre-planning in the within-object case. For example, “between-object saccades compensated for the displacement to aim for a target position on the new object whereas within-object saccades did not show compensation”. If there is separate planning and intentionality for both saccades, then each fixation may be informative for processing the observed object. Nevertheless, we chose the conservative analysis route and combined successive fixations on the same region. The results would have been strengthened if we had not combined fixations. 

An increase in number of fixations is of course inconsistent with the presence of a mechanism of the type proposed to explain the alternative outcome (of fewer fixations on detected cards), i.e., that detection is a result of perceiving an inherent property of the detected target and that once this property is perceived detection is immediate. Rather, our findings suggest that several stages are involved in the perceptual process leading ultimately to target pair detection. 

Similarly, our results rule out the hypothesis that fixations are irrelevant for detection. This alternative might stem from a need for perceiving a particular card aspect, triggered in random fashion by a single fixation. On average, the number of fixations on the two pairs would be equal, and chance would determine which pair is detected first. 

An inherent requirement for performance of the Identity Search task is the use of Working Memory to enable comparison of different cards. Ballard, Hayhoe, Pook, and Rao (1997) suggest a tradeoff between working memory load and the number of required fixations. The well-known limitations of Working Memory (Horowitz & Wolfe, 1998; McCarley, Wang, Kramer, Irwin, & Peterson, 2003) may make repeated fixations to the same card advantageous for this comparison process. Beyond their deictic role suggested by Ballard et al. (1997), repeated fixations can be used to gather information and integrate it. 

The greater number of fixations on the ultimately detected pairs raises an essential issue: What comes first? Does an arbitrary intensive observation of the pair cards lead to perception, or, the opposite, does detection of the pair lead to more fixations on it? In other words, does the larger number of fixations give rise to detection or is it a result of detection? 

To discriminate between these possibilities and determine which, if any, is the actual relationship between fixation and detection, we analyzed the sequence of fixations to determine when the differentiation between detected and undetected cards occurs. We argued that if fixations were randomly organized, and chance led to more fixations on one pair than on the other—and thus to its ultimate detection—then there should be no regular pattern of fixations on the detected cards until the moment of detection. Any regular pattern should derive from a change of state in the search sequence—that is, from some perceptual detection mechanism. 

On the other hand, if it is detection that leads to increased fixations, the period of increased fixations on the detected cards should be very brief. Once there is conscious recognition of the existence of the pair, there should be very few fixations before the ultimate marking of these cards. Note that the time taken for confirmation that detected cards indeed form an identical pair is probably very brief. We conclude this from another experiment that we performed, with a much more complex task, the Set game (Jacob & Hochstein, 2008 and unpublished results). We asked subjects to verify whether the few cards presented (for only 300 ms) form a three-card Set. They achieved 96% accuracy. This speed should be at least matched in the easier two-card Identity Search task. Thus, the confirmatory check period should have required no more than the time taken for 1–2 fixations. Another indication of the ease of confirmation is that sometimes pairs are detected after very few fixations (Figure 9). An additional indication that pure confirmation of cards was not conducted during the period of increased fixations on the detected pair comes from the slope in Figure 8b. If final fixations reflected consistent confirmation, the slope during these last fixations would have had the value of 1 (indicating that the next fixation was always on a pair card), which is not the case. We conclude that subjects are not yet consciously aware of the target pair at this point—though the visual system may already have some information in this direction, leading to the preponderance of such fixations. 

There is additional evidence of a mismatch between fixation and detection, that is, that fixation on the target is not always accompanied by explicit detection (Barlasov-Ioffe & Hochstein, 2008; Motter & Belky, 1998; Rutishauser & Koch, 2007; Sheinberg & Logothetis, 2001). Recognition often requires a “double-take” saccade, i.e., one or more fixations away from the target (Rutishauser & Koch, 2007), during which conscious recognition presumably occurs, leading the eyes back to the target. We suggest that in the period between return fixations, even when awareness of the target is at most unconscious—since the response is still not initiated—one should not consider the mind as being completely ignorant, but as having implicit pre-recognition. This implicit pre-recognition stage may be similar to that found in the persistent activity of temporal cortex (Sheinberg & Logothetis, 2001) preceding target acquisition, and without target awareness. We suggest that this implicit recognition guides the eyes and saccadic planning. 

We turn to the backward dynamics, the accumulated number of fixations on pair cards, averaged over trials, with respect to the moment of card marking, and demonstrated in Figure 8. The results show that the patterns of fixations on the detected and undetected pairs are nearly identical up to a point a few fixations before marking. At this point, there is a sharp upturn in the number of fixations on the detected pair. 

The dynamics are not very different for short compared to long trials (see Figure 9). The small increase in cumulative number of fixations on detected pair cards in the long search sequences (6.4 vs. 3.2 fixations), relative to the total number of fixations in the sequence, implies that the number of fixations on pair cards needed for identification is defined within a certain period of time—that is, there is a dying memory type of recall and comparison. 

Nevertheless, longer searches, i.e., longer sequences of fixations, naturally raise the total number of fixations, and therefore in particular also raise the number of fixations on the pair cards. Thus, the important factor determining detection must take into account also the proximity of fixations. We conclude that not only the absolute number of fixation has an influence but also their proximity in time or number of intervening fixations. 

We suggest that since fixations on each pair are sparse during the initial part of the search, the fixation slope ( Figure 8b) is still low. Only at a later stage do the fixations on the pair card become closer to each other and the slope rises. 

We found that the sequential distance between fixations were different for detected and undetected pair cards suggesting that not only the number of fixations but also the time between them is a significant factor for determining pair detection. This factor may be related to the limited capacity of Working Memory (Horowitz & Wolfe, 1998; McCarley et al., 2003; Phillips & Christie, 1977a, 1977b). It may be difficult for subjects to keep many cards in Working Memory at the same time, so that fixations need to be close to each other to associate place with identity. The average sequential distance (Figure 5) perhaps puts an upper bound on average working memory at 4 cards. The sequential distance decreases when approaching detection (Figures 6d and 7d), so that this may even be an overestimate (see also the recency effect with free viewing in Phillips & Christie, 1977a, who used a similar stimulus pattern but different paradigm). Perhaps a necessary condition for detection is that two cards be represented concurrently in Working Memory. 

Still, even viewing two cards one right after the other does not ensure that they be recognized as an identical pair ( Figure 5b). We conclude that no single sequential-distance fixation pattern (fixating the pair cards one after the other, with one card intervening, etc.) is sufficient by itself to bring about detection, nor is any one pattern necessary. Nevertheless, very close patterns (with a sequential distance of 1–3) are more prevalent for the detected cards implying that perhaps one or the other of these is a necessary condition for detection. Perhaps a conjunction of a few patterns is sufficient for determining detection. 

We suggest that the important factor may be the change in the accumulated number of fixations, i.e., the slope, rather than the number itself. Detection may depend on the slope rising above some threshold. The slope does not rely on the intersection point between the two graphs (of the detected and undetected pairs— Figure 8a). The slope is equivalent to the probability that the next fixation will be on a pair card. Its rise reflects an end of random, unproductive search. The point where the slope exceeds a pre-determined threshold may be regarded as the bifurcation point where there is a change of state in the search process. 

This bifurcation point may reflect a transition between a first stage of “search in the dark” to a second stage of “early implicit recognition”. This suggests that there is an early recognition stage, which is followed by more fixations. 

The finding that there is a bifurcation point a while before the time of marking the cards suggests that there might have been implicit perception of the pair before its entering into conscious awareness (Mitroff, Simons, & Franconeri, 2002; Rensink, 2004). This implicit perception may direct eye movements to the pair cards; however, note that not all fixations are on these cards even at this point. During this stage of concentrating on the pair cards, the unconscious discovery is brought to awareness. 

In summary, we suggest a 3-stage model of the perceptual recognition process, as follows: Stage 1: Initial search—random fixations on the different cards in arbitrary order—a “search in the dark”. Stage 2: Implicit (unconscious) recognition of the target pair, perhaps controlling and guiding eye movements to the relevant sensed location of these target cards. The transition from Stage 1 to Stage 2 is seen in the bifurcation point of the fixation slope in Figure 8. Stage 3: Insight: Explicit detection with conscious knowledge of target presence and its location—followed by rapid marking of the two cards. 

Conclusions

Searching for a pair of identical cards, where two such pairs were present in each display of twelve cards, the cards of the pair that was ultimately detected are observed more frequently than cards of the undetected pair—there are more fixations and longer fixations on the ultimately detected pair, and the average sequential distance between fixations on card regions is smaller for the detected pairs. 

A bifurcation point is observed along the dynamics of search, in which the to-be-detected pair overpowers the undetected one. This suggests an early, implicit, recognition stage in the process of perception, which is followed by more fixations, leading, ultimately, to the point of explicit target pair recognition. 

Acknowledgments

We thank Anne Treisman and Robert Shapley for helpful discussions throughout this study and Daphna Weinshall and Leon Deouell, members of the MJ doctoral committee, for comments and suggestions. We thank Michael Wagner and Guy Goldner for assistance with the eye-movement recording equipment. 

This study was supported by grants from the Israel Science Foundation (ISF) and the US–Israel Binational Science Foundation (BSF). We are grateful to the National Institute for Psychobiology in Israel and Prof. Micha Spira for use of the Charles E. Smith and Joel Elkes Laboratory for Collaborative Research in Psychobiology for the eye-movement recording reported here. 

Commercial relationships: none. 

Corresponding author: Michal Jacob. 

Email: [e-mail]. 

Address: Department of Neurobiology, Institute of Life Sciences, Hebrew University, Givat Ram, Jerusalem 91904, Israel.