Faces in the eye of the beholder: Unique and stable eye scanning patterns of individual observers

Abstract

Abstract

Abstract:

Abstract  Eye scanning patterns while viewing pictures have provided valuable information in many domains of visual cognition. Such patterns are determined by the type of image viewed (e.g., faces, scenes) as well as the task individuals are asked to perform (e.g., visual search, memory). Here we show that another key factor that significantly influences eye scanning patterns but has been mostly overlooked is the individual observer. During face viewing, we found that individuals showed diverse scanning patterns that, in many cases, were inconsistent with the typical triangular shape pattern that is commonly observed when eye scanning patterns are averaged across individuals. These idiosyncratic eye scanning patterns were not random but highly stable even when examined 18 months later. Interestingly, these eye-tracking patterns were not predictive of behavioral performance. Such stable and unique scanning patterns may represent a specific behavioral trait/signature and be formed early in development, reflecting idiosyncratic strategies for performing visual recognition tasks.

Introduction

While the eyes are considered a window to the soul (Hess & Polt, 1960), eye movements offer a glimpse into cognition (Just & Carpenter, 1976). Tracking eye position has provided insights into many domains, including reading (Rayner, 1998), arithmetic (Suppes, 1990), picture scanning (Noton & Stark, 1971), human–computer interaction (Jacob & Karn, 2003), and driving (Land & Lee, 1994). Broadly, eye movements are categorized into fixations, during which the eye remains steady on a specific location, and saccades, which are movements of the eyes between fixations (Salvucci & Goldberg, 2000). The location of fixations is considered the locus of overt attention (Wright & Ward, 2008) and has been used to determine where information necessary for performing a visual task resides. 

One of the most well-established findings in both the study of eye movements and in the study of face perception is the typical triangular scanning pattern that individuals show when they view faces (Figure 1). This seminal finding was first reported by Yarbus (1967) and still appears in most textbooks on cognitive sciences. This typical scanning pattern, which has been replicated many times (Althoff & Cohen, 1999; Arizpe, Kravitz, Yovel, & Baker, 2012; Blais, Jack, Scheepers, Fiset, & Caldara, 2008; Hsiao & Cottrell, 2008; Peterson & Eckstein, 2012; Williams & Henderson, 2007), suggests that eye movements are determined by the structure and content of the stimulus we view. For faces in particular, people tend to focus on the internal facial features, primarily the eyes but also the nose and mouth. Notably, these common findings are derived from the average scanning patterns across individuals. 

Figure 1

 

Typical triangular-shape eye scanning patterns across the internal facial features. (A) Data originally reported in the seminal study by Yarbus (1967). (B) Eye scanning patterns from two more recent studies (Arizpe et al., 2012; Blais et al., 2008) show similar triangular patterns.

In addition to the type of stimulus, a second factor that has been shown to influence eye movements is task demands; asking observers different questions about a picture of a scene or a face produces different scanning patterns (Armann & Bulthoff, 2009; Malcolm, Lanyon, Fugard, & Barton, 2008; Walker-Smith, Gale, & Findlay, 1977; Yarbus, 1967). For example, Malcolm and colleagues (2008) have shown that when subjects were asked to match the identity of two faces, they fixated more on the upper than the lower part of the face. In contrast, when they were asked to match facial expression, they fixated more on the lower than the upper part of the face. However, a recent study that applied a classifier approach failed to find effects of task factors on eye movements (Greene, Liu, & Wolfe, 2012). 

Although these findings emphasize the role of stimulus and task in determining eye scanning patterns, in the current study, we report that eye scanning patterns are greatly influenced by another important factor, which has so far been mostly overlooked: the individual observer (however, see Peterson & Eckstein, 2013). To investigate the contribution of the individual observer to variation in eye scanning patterns, eye movements were recorded while subjects performed a face recognition task. To assess how stable the eye scanning patterns of each individual are across time, three study test sessions were performed on three different days: Day 1, Day 3, and 18 months later. 

Materials and methods

Participants

Eighteen Caucasian undergraduate students of the psychology department at Tel Aviv University participated in this experiment. Subjects were given academic credit for their participation. All the participants gave written informed consent, and the protocol was approved by the ethics committee. Nine of the original participants agreed to be retested 18 months after the original examination and were paid $10 for their participation. 

Stimuli

Twenty-four Caucasian faces were taken from the Harvard face database. All images are gray scale, front view, evenly lit photos of real men wearing black knit hats covering their hair. Images were scaled to roughly 400 pixels wide by 500 high, subtending 15.8° of visual angle horizontally and 19.5° of visual angle vertically, and placed on the center of a 1,024 by 768 pixels white background. The average pixel brightness of all the faces (hats excluded) was 139.5, and the standard deviation was 32.7. 

Behavioral task

Subjects performed an old/new face recognition task. Twelve faces were presented during a study phase during which subjects were asked to memorize the faces. Each face was preceded by a 1000-ms fixation dot. Then the face was presented for 750 ms followed by a 530-ms interstimulus interval. The test phase was initiated by the experimenter after the completion of the study phase. Subjects were presented with instructions on the computer screen that indicated which key they needed to press for old or new faces. After reading the instructions, the subjects pressed a key, which initiated the presentation of 24 faces, 12 of which were presented in the study phase and 12 new faces. Each trial started with a 1000-ms fixation point followed by a face presented for 1250 ms. Subjects were asked to press one key for old faces and another key for new faces. Eye-tracking data collection was terminated if a response was made before 1250 ms. The next trial started after the subject made the key press. To avoid starting-location biases (Arizpe et al., 2012), fixations were presented either to the left or the right prior to the centrally presented face. The task lasted approximately 3–5 min. 

Eye tracking

Eye-tracking data were collected during the study and test phases. An SMI Red eye-tracker set to a 60-Hz sampling rate was used. The eye-tracker was positioned beneath a 19-in. monitor, connected with a DVI cable to the stimulus presentation computer. Eye-tracking data were recorded on a dedicated SMI computer. 

A chin rest was positioned approximately 55 cm from the monitor, and subjects were instructed to use it throughout the entire recording. Calibration and validation were run before the study test on each day. Each calibration and validation consisted of nine points presented in random order on the monitor. Whenever validation failed to attain an error of below 1°, calibration was run again until the threshold was met. Calibration screens were presented with the Python programmed experiment environment, using the PyGame library to display images and capture keystrokes. 

Data analysis

Data processing

Analysis was based on raw eye samples. This was done in order to get the maximal data available from the eye-tracker during the short trials, indeed while sacrificing noise levels. Data samples were filtered for invalid on-screen pixel locations, nonpositive pupil diameter, and nonpositive pupil confidence. 

Stimulus alignment

Stimulus images were presented in the same place on the monitor but were not perfectly aligned. To allow for accurate processing of the eye-tracking data, recorded samples were retroactively aligned according to the actual stimulus presented. Each stimulus image was manually labeled with nine landmark points: inner and outer eye corners, center of the pupil, tip of the nose, and corners of the mouth. An average location for each of the nine points was calculated across all stimuli; then, for each stimulus, a best-fit transform was found. The transform was allowed 4° of freedom: horizontal and vertical translation and horizontal and vertical scaling. The transform minimizes the sum of square distances between each of the stimulus' nine landmark points and their corresponding average locations. This transform was applied to the recorded data samples according to the stimulus being viewed. This resulted in data being scaled 3.7% and 3.8% (up or down) on average on the x- and y-axis, respectively, then translated 20 and 21 pixels on average on the x- and y-axis (see Supplemental material Figure 3S for analysis of unaligned faces). 

Eye movement pattern similarity measures

The similarity measure is based on the normalized cross-correlations between the heat maps of two sets of trials. First, a heat map was generated for each set (see Figure 2). Each cell in the heat map matrix represents the cumulative number of data points present in the cell's coordinates. Each heat map was scaled down by a factor of two in each dimension to increase computation speed and then smoothed with a Gaussian filter (kernel standard deviation of 7 pixels or 0.6° of visual angle). Normalized cross-correlation was used to attain a similarity value between the two heat maps. Each map is treated as a single vector, and the desired value is the Pearson's r value, the correlation between these two vectors. Therefore, similarity ranges between one for two blocks with heat maps identical in pattern and zero for two blocks with uncorrelated heat maps. Negative values are theoretically possible but rarely emerge in practice. It is important to note that this method for comparing gaze patterns ignores the order of the samples across time. Only the location of the data samples is taken into account. 

Figure 2

 

Eye-gaze density maps depict eye-gaze location during a study and a test phase of a face recognition task. Each row depicts one individual. Each column depicts a different testing session. On each of the three days, subjects had to memorize a face during the study phase, which was followed by a test phase, during which subjects indicated whether they saw a new or an old face. Data show striking similarities within individuals even across a period of 18 months.

To estimate the statistical significance of our findings, the multiresponse permutation procedures (MRPP) (Mielke, Berry, & Johnson, 1976) were used. A single heat map was generated for each block of trials of either the trials presented in the study phase or trial presented in the test phase for each subject. Study-phase trials and test-phase trials were analyzed separately because differences across the two phases may reflect the different exposure duration of the stimulus. Our analysis was therefore focused on assessing similarity within the study or within the test phase across sessions (within subjects) versus across participants (between subjects). MRPP was adapted to test whether a given partition of blocks into groups of blocks formed a nonrandom structure. A nonrandom structure is a clustering in which similarities of blocks of trials belonging to the same group are large and similarities between blocks of trials of different groups are smaller. The significance of the clustering is measured relative to a random clustering of the block of trials into groups with the same size as the proposed clustering. 

Following Good (1994), let ω₁ … ω_n designate n blocks, δ_ij the similarity between block i and j as defined above, S₁ … S_g represent an exhaustive partitioning of the n blocks into g disjoint groups, and n_j be the number of blocks in the jth group. 

Define the average between-blocks similarity for all blocks within the ith group,  

 

 

where φ_i(ω) is an indicator function that is one if Display Formula and zero otherwise. The test statistic is the weighted within-group average of these distances,  

 

 

The permutation distribution of Δ is taken over all allocations of the n blocks into g groups with the same number of blocks in each group. The significance of the clustering is the percentage of random clusters with Δ bigger or equal to the Δ calculated according to the true clustering. In practice, the distribution of Δ was approximated with 10,000 random permutations. 

When analyzing eye-tracking data, it is important to consider equipment calibration bias. Each eye-tracker calibration imposes an inherent and unavoidable bias in the data. The bias is a result of the error term in the calibration process. Because a calibration is performed once per session in order to compare between-subject to within-subject patterns, the patterns within subjects should be taken from different sessions to assure that higher within-subject versus across-subject similarity is not due to the effects of calibration. 

Relationship between eye-tracking pattern and performance level

To investigate the relationship between eye movements and response accuracy, the Mantel test was used (Mantel, 1967). Existence of such a relationship means that blocks with similar eye-movement patterns also exhibit similar response accuracy. The analysis is based on a dissimilarity measure, which is defined as one-similarity. Two distance matrices, A and B are defined. The first matrix represents the distance (dissimilarity) of eye movements; A_ij is the dissimilarity value between the ith block's heat map and the jth block's heat map. The second matrix represents the distance in performance; B_ij is the absolute difference between the response accuracy in the ith block and the jth block. With this framework, a correlation between eye movements and response accuracy means that corresponding elements of the two matrices vary in coordination. 

In addition, to specifically test whether performance level on a face-recognition task is better when individuals fixate on the upper than the lower part of the face (Malcolm et al., 2008; Peterson & Eckstein, 2012), we correlated performance level with the average fixation location along the vertical axis of the face. To confirm these findings, the same analysis was conducted on a second group of 29 participants performing a highly similar task in the laboratory on Brain and Cognition at the National Institute of Mental Health (see Methods and results in Supplementary Material). 

Results

Table 1 specifies the mean and standard deviation of performance level (d′) on the face memory task for the 18 subjects who participated in two sessions and for the subset of nine subjects that participated in three sessions. There were no significant differences in performance level across the two sessions for the 18 subjects, t(17) = 1.68, p = 0.10, and across the three sessions for the nine subjects, F(2, 8) = 1.25, p = 0.3. 

Table 1

 

Performance level on the face memory task.

	Day 1	Day 3	18 months
18 subjects	2.56 (1.22)	1.81 (1.25)
Nine subjects	2.55 (1.13)	2.09 (1.49)	1.81 (0.70)

Figure 2 shows eye scanning patterns during the study and the test phase of a face recognition task across the three testing sessions. Consistent with previous studies, averaging eye scanning patterns across individuals revealed the typical triangular-shape scanning pattern across the internal facial features (Figure 2, last row). Interestingly, however, examination of each individual's eye movements revealed distinct scanning patterns that, in many cases, were not consistent with the average triangular shape. Figure 2 shows eye-gaze density maps for the study and test phases of the nine participants who participated in all three sessions (Supplementary Figure S1 shows the same results for all 18 participants who participated in the first two sessions; see also Supplementary Figure S4 for an additional sample of 29 participants). The data reveal that eye-tracking patterns vary vastly across individuals, but each individual shows a remarkably consistent pattern throughout the different testing sessions even 18 months later. Interestingly, the average eye-tracking pattern (Figure 2, bottom row) clearly does not reflect the individual strategies and the great variance among them. The average eye-tracking pattern for each individual across sessions (Figure 2, rightmost column) is very similar to the pattern revealed for each of the separate sessions, indicating the highly stable pattern within individuals across sessions. 

To quantify the pattern of eye movements, we generated a similarity matrix between all study test phases in the three time points for the nine subjects who participated in the three sessions (Figure 3, see Supplementary Figure S2 for the same matrix for two sessions for all 18 subjects). Each cell represents the similarity in the eye scanning patterns between two heat maps both within and across different subjects. The plot displays the same main findings we see in Figure 2; within-subjects' squares along the diagonal are hot, meaning subjects usually displayed consistent eye-tracking patterns across sessions, but most subjects are distinct from other subjects. 

Figure 3

 

Similarity matrix indicates the cross-correlations of heat maps (shown in Figure 2) among the scanning patterns in the study and test phases on different days (Day 1, Day 3, 18 months) across and within individuals. Data show greatest similarity within individuals (big square along the diagonal—see example of one subject on the right) across the different sessions and no evidence for a scanning strategy that is common to all individuals. Subjects are ordered according to the similarity among them.

To statistically estimate the similarity across tasks and subjects, we performed MRPP (for details, see Materials and methods). First, to test similarity in eye-tracking patterns across participants, the average similarity between each subject's Day 1 and Day 3 eye-gaze density maps was compared against the average similarity when randomly pairing Day 1 and Day 3 maps of different subjects. Analysis was performed separately for the study and the test sessions. The permutation test revealed highly distinct patterns across the 18 participants who participated in two sessions both for the study phase (p < 0.00001) and the test phase (p < 0.00001). Similar analysis for the subset of nine subjects who participated in three sessions, comparing eye-gaze density maps on Day 1 and 18 months later, also revealed highly distinct patterns across individuals for the study (p = 0.004) and test (p = 0.003) phases. These data clearly show that these idiosyncratic eye-tracking patterns are not random but highly stable over time. 

Second, MRPP was also used to investigate the effect of “session” across the 18 subjects who participated in two sessions. The test was performed separately for each phase (study/test). For each phase, two heat maps were generated per subject: Day 1 and Day 3. If across subjects there is an eye scanning pattern that is specific for Day 1 versus Day 3, then Day 1 heat maps will be similar to each other, and Day 3 heat maps will also be similar to each other, but Day 1 heat maps will be distinct from Day 3 heat maps. However, the average interpoint similarity within each day was not greater than the average interpoint similarity when randomly labeling the Day 1 and Day 3 heat maps within each subject, neither for the study phase (p = 0.85) nor for the test phase (p = 0.27). Similar analysis for the nine subjects who participated in the three sessions shows that the three sessions are indistinguishable as well, for both the study (p = 0.77) and test (p = 0.34) phases. (See Supplementary Material for single trial classification analysis.) 

Given the highly distinct eye scanning patterns across individuals, we asked whether particular eye-tracking patterns may be associated with better or worse face recognition abilities across individuals. Figure 4 shows eye density maps sorted by performance level on the face memory task. As can be seen, there is no specific location on the face that is associated with better or worse performance. For example, subjects who primarily fixated on the mouth show highly different performance on the task from the second best (d′ = 3.11) to the poorest recognition level (d′ = 0.42). To estimate the relationship between eye scanning patterns and performance levels, Mantel tests (see Materials and methods) were used. The test was performed for 18 subjects on each of the two sessions they participated in and revealed no significant relationship between eye-tracking pattern and performance level (Day 1, p = 0.1; Day 3, p = 0.17). Similar analysis restricted to the nine subjects who participated in three sessions also revealed no relationship between performance and eye-tracking patterns (Day 1, p = 0.57; Day 3, p = 0.75; 18 months, p = 0.43). 

Figure 4

 

Eye-gaze density maps depicting eye scanning patterns sorted based on performance level on the face recognition task. Lower panel shows the lack of association between performance level and average location of eye gaze density along the y-axis.

To further assess whether better performance level is associated with fixations on the upper than the lower part of the face, we computed a correlation of performance level on the face recognition task with the mean eye-gaze density along the y-axis reveal (see scatterplot in Figure 4). Data indicate no significant correlation (r = −0.19). To assure that these findings do not specifically reflect the current sample, we examined the relationship between performance and eye scanning patterns in another sample of 29 subjects who performed a face recognition task (see Supplementary Figure S4). Correlation across performance level and the average eye-gaze density along the y-axis revealed no relationship between the two measures (r = 0.12). 

Discussion

Our study reveals that eye scanning patterns vary greatly across individuals. Importantly, many individuals do not show the typical triangular-shape eye scanning pattern across the internal facial features (see Figure 1) that has been repeatedly reported previously. Moreover, these interindividual diffences in eye-movement patterns were not random but remarkably stable over a long period of time, suggesting that different individuals employ their own idiosyncratic eye scanning strategy to complete a visual recognition task. 

Whereas the majority of eye-movement studies have focused on the average of various eye-movement parameters across individuals (Althoff & Cohen, 1999; Arizpe et al., 2012; Blais et al., 2008; Hsiao & Cottrell, 2008; Peterson & Eckstein, 2012; Williams & Henderson, 2007), few prior studies have reported individual differences in occulomotor aspects of eye movements, such as fixation durations, number of fixations, or amplitude of saccades (Andrews & Coppola, 1999; Boot, Becic, & Kramer, 2009; Castelhano & Henderson, 2008; Rayner, Li, Williams, Cave, & Well, 2007; Sekiguchi, 2011). For example, Greene and colleagues (2012) recently showed that a simple classifier trained on these eye-tracking parameters of each individual could successfully classify participants across different scene-processing tasks. However, these studies did not examine whether these individual differences are stable across a long period of time or investigate the spatial patterns of eye movements. Although stability across time was recently reported for the location of the first fixation of briefly presented faces, this was largely attributed to the short presentation of the stimulus (Peterson & Eckstein, 2013). Here we clearly show that when faces are presented for more than 1 s and all data points are considered, including multiple fixations, remarkable consistency in scanning patterns is found even over time periods as long as 18 months. These findings indicate that intrinsic and idiosyncratic characteristics of each individual strongly determine scanning patterns to visual stimuli in addition to the effects of stimulus and task parameters. Idiosyncratic strategies of eye scanning during face recognition were also reported by Miellet, Caldara, and Schyns (2011), who showed that individuals may use different eye scanning strategies across different trials when performing a face recognition task. These findings are consistent with our findings in that they show that different eye scanning strategies may be as effective for successful face recognition. They add to our findings by showing that under certain circumstances (e.g., familiar faces, gaze-contingent task) the same subject may use different strategies across different trials. Our findings cannot be directly compared to these findings as we presented unfamiliar faces and did not use a gaze-contingent task, which may influence the strategy individuals use to perform that task. 

Interestingly, the wide variation among individuals in eye scanning patterns did not predict performance on the face recognition task. Most surprisingly, individuals who primarily fixated on the mouth area did not show worse performance than individuals who primarily fixated on the eye area, which has been considered as the diagnostic location for face recognition (Malcolm et al., 2008; Peterson & Eckstein, 2012; Schyns, Bonnar, & Gosselin, 2002). The lack of association between eye scanning patterns and performance on the face recognition task implies one of two possible alternatives. First, that there is no one diagnostic location on the face that is associated with efficient face recognition common to all individuals (Arizpe et al., 2012). This conclusion, however, is inconsistent with many studies that point to a specific location—the eye region—as the most diagnostic area for face recognition (Caldara, Zhou, & Miellet, 2010; Peterson & Eckstein, 2012; Schyns et al., 2002). It is also inconsistent with studies on pathological populations, such as prosopagnosic or autistic individuals who show face recognition deficits as well as more frequent fixations on the mouth rather than the eye area (Caldara et al., 2005; Kliemann, Dziobek, Hatri, Steimke, & Heekeren, 2010; Pilyoung et al., in press; Van Belle et al., 2011). Alternatively, the location where participants are fixating during face recognition may reflect an interaction between idiosyncratic strategies individuals employ combined with stimulus and task factors and therefore cannot directly indicate where the diagnostic information that is critical for recognition resides. This latter suggestion is consistent with recent studies that examined eye-tracking patterns in individuals from different races. These studies show that Asian and Caucasian observers fixate at different location on the face regardless of the race of the face (Blais et al., 2008; Caldara et al., 2010; but see Goldinger, He, & Papesh, 2009). Specifically, Asians fixate on the nose more than the eye area whereas Caucasians focus more on the eye area. Furthermore, given that this strategy was similar for both their own race and faces of other races, despite the fact that recognition of other race faces is poorer than one's own race faces, further suggests that specific patterns of eye tracking may not be directly associated with performance level on a task in the normal population. Our findings are also consistent with a recent study showing that individuals who tend to focus on the mouth area had worse performance when they were forced to fixate on the eye area rather than the mouth area (Peterson & Eckstein, 2013). More generally, a lack of association between the location of eye gaze and task performance in the normal population (Blais et al., 2008; Peterson & Eckstein, 2013; Sekiguchi, 2011) and its clear existence in pathological populations suggests that the pathological and the normal populations reflect two separate populations that show qualitatively rather than quantitatively different behaviors. 

Whereas the current data undoubtedly show very large and highly stable individual differences in the spatial distribution of fixations across a very long period of time, many questions remain open. Do these individual differences start early in development? A recent study showed that cultural differences in eye-tracking patterns are observed already at the age of seven (Kelly et al., 2011), suggesting that the different strategies individuals are using may be evident at an early stage in development. If indeed the eye region is the diagnostic area for face recognition, do mouth fixators show better parafoveal vision? Are different personality traits or developmental abnormalities associated with different fixation locations? Finally, given that studies do report effects of stimulus and task on eye-tracking patterns (e.g., Malcolm et al., 2008), how do individual's characteristic scanning patterns interact with task and stimulus factors? 

In conclusion, our data show very large and highly stable individual differences in eye scanning patterns during a face recognition task. These findings highlight that, in addition to task and stimulus factors, fixation patterns are highly influenced by each individual's characteristic and idiosyncratic scanning patterns. These data also suggest that the well-known T shape or a triangle shape of fixation pattern that covers the eye, nose, and mouth areas of the face reflects the average map across individuals (Figure 1 and Figure 2, last row) but does not correspond to scanning patterns of many individuals. Finally, the location of fixations may not necessarily reflect where the diagnostic information needed for recognition resides, and other techniques that allow more restricted viewing, such as the bubble (Schyns et al., 2002) or the gaze-contingent procedures (Caldara et al., 2010), should be used to complement free-viewing data in determining what visual information that is important for recognition is located. 

Supplementary Materials

docx S1 

Acknowledgments

This work was supported by a binational science foundation (BSF) grant to G. Y. and C. I. B. C. I. B. and J. A. are supported by the Intramural Research Program of the National Institute of Mental Health. Author contributions: E. M. collected data, performed data analysis, and wrote the paper; J. A. collected data and performed data analysis; C. I. B. designed the study and wrote the paper; G. Y. designed the study and wrote the paper. 

Commercial relationships: none. 

Corresponding author: Galit Yovel. 

Email: [e-mail]. 

Address: School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel.