Face exploration dynamics differentiate men and women

Abstract

The human face is central to our everyday social interactions. Recent studies have shown that while gazing at faces, each one of us has a particular eye-scanning pattern, highly stable across time. Although variables such as culture or personality have been shown to modulate gaze behavior, we still don't know what shapes these idiosyncrasies. Moreover, most previous observations rely on static analyses of small-sized eye-position data sets averaged across time. Here, we probe the temporal dynamics of gaze to explore what information can be extracted about the observers and what is being observed. Controlling for any stimuli effect, we demonstrate that among many individual characteristics, the gender of both the participant (gazer) and the person being observed (actor) are the factors that most influence gaze patterns during face exploration. We record and exploit the largest set of eye-tracking data (405 participants, 58 nationalities) from participants watching videos of another person. Using novel data-mining techniques, we show that female gazers follow a much more exploratory scanning strategy than males. Moreover, female gazers watching female actresses look more at the eye on the left side. These results have strong implications in every field using gaze-based models from computer vision to clinical psychology.

Introduction

Our eyes constantly move around to place our high-resolution fovea on the most relevant visual information. Arguably, one of the most important objects of regard is another person's face. Until recently, a majority of face perception studies have been pointing to a “universal” face exploration pattern: Humans systematically follow a triangular scanpath (sequence of fixations) over the eyes and the mouth of the presented face (Vatikiotis-Bateson, Eigsti, Yano, & Munhall, 1998; Yarbus, 1965). However, more recent studies found that face-scanning strategies depend upon many factors, including task (Borji & Itti, 2014; Borji, Lennartz, & Pomplun, 2015), social context (Foulsham, Cheng, Tracy, Henrich, & Kingstone, 2010; Gobel, Kim, & Richardson, 2015), emotion (Eisenbarth & Alpers, 2011; Schurgin et al., 2014), personality (Perlman et al., 2009; Peterson & Eckstein, 2013), and even culture (Blais, Jack, Scheepers, Fiset, & Caldara, 2008; Wheeler et al., 2011). For instance, when watching videos of faces, observers believing that the targets would later be looking at them looked proportionally less at the eyes of the targets with higher ranked social status (Gobel et al., 2015). Other studies showed that when participants discriminate between emotional and neutral facial expressions, distinct fixation patterns emerge for each emotion. In particular, there is a focus on the lips for joyful faces and a focus on the eyes for sad faces (Schurgin et al., 2014). In parallel, it has very recently been shown that humans have idiosyncratic scanpaths while exploring faces (Kanan, Bseiso, Ray, Hsiao, & Cottrell, 2015) and that these scanning patterns are highly stable across time, representing a specific behavioral signature (Mehoudar, Arizpe, Baker, & Yovel, 2014). In the latter study, the authors asked the subjects to perform a face recognition task during three test sessions performed on three different days: Day 1, Day 3, and 18 months later. Their results show that individuals have very diverse scanning patterns. These patterns were not random but highly stable even when examined 18 months later. 

In this study, we aimed to identify which factors drive this idiosyncratic behavior. To quantify their relative contributions, we had to overcome two major difficulties. First, in order to take into account as many factors as possible, we needed an unprecedentedly large eye-tracking data set. We had to move beyond the usual small-sized eye-tracking data sets with restricted participant profiles, the famous WEIRD (western, educated, industrialized, rich, and democratic) population (Henrich, Heine, & Norenzayan, 2010). To do so, we increased our participant pool by recording and exploiting the largest and most diverse eye-tracking data set that we are aware of (405 participants between 18 and 69 years old, 58 different nationalities; Winkler & Subramanian, 2013). Second, we needed to equip ourselves with eye-tracking data-mining techniques that could identify and quantify eye-gaze patterns in the data set. The vast majority of previous studies quantified observers' scanpaths through spatial distributions of eye positions averaged over time, deleting the—manifestly critical—temporal component of visual perception (Le Meur & Baccino, 2013). In contradistinction, we proposed a new data-driven approach able to encapsulate the highly dynamic and individualistic dimension of a participant's gaze behavior. We identified systemic differences that allowed a classifier solely trained with eye-tracking data to identify the gender of both the gazer and actor with very high accuracy. 

Methods and results

Experiment

This data set has been described and used in a pupillometry study (Binetti, Harrison, Coutrot, Johnston, & Mareschal, 2016). Eye data, stimuli, and demographics are available at http://antoinecoutrot.magix.net/public/databases.html. 

Participants

We recorded the gaze of 459 visitors to the Science Museum of London, UK. We removed from the analysis the data of participants under age 18 (n = 8) as well as 46 other participants whose eye data exhibited some irregularities (loss of signal, obviously shifted positions). The analyses are performed on a final group of 405 participants (203 males, 202 females). Mean age of participants 30.8 years (SD = 11.5; males: M = 32.3, SD = 12.3; females: M = 29.3, SD = 10.5). The experiment was approved by the UCL research ethics committee and by the London Science Museum ethics board, and the methods were carried out in accordance with the approved guidelines. Signed informed consent was obtained from all participants. 

Stimuli

Stimuli consisted of video clips of eight different actors (four females, four males, see Figure 1). Each clip depicted the actor initially gazing toward the bottom of the screen for 500 ms, then gazing up at the participant for a variable amount of time (between 100 and 10,300 ms, in 300-ms increments across 35 clips), and finally gazing back at the bottom of the screen for 500 ms. Actors kept their head still and maintained a neutral facial expression. Width × Height = 428 × 720 pixels (16.7° × 28.1° of visual angle). Faces occupied most of the image, on average 280 × 420 pixels (10.9° × 16.4° of visual angle). Average size of the eyes was 75 × 30 pixels (2.9° × 1.2° of visual angle), nose 80 × 90 pixels (3.1° × 3.5° of visual angle), and mouth: 115 × 35 pixels (4.5° × 1.4° of visual angle). Frame rate = 30 Hz. Videos were shot with the same distance between the actors and the camera in the same closed room with no window in diffuse lighting conditions. Actors sat against a green background, and the point between their eyes (nasion) was aligned with the center of the screen. Hence, the position of facial features slightly varied between actors due to individual morphological differences, but were largely overlapping between actors. 

Apparatus

The experimental setup consisted of four computers: two for administering the personality questionnaire and two dedicated to the eye-tracking experiment and actor face-rating questionnaire (see Procedure). Each setup consisted of a stimulus presentation PC (Dell precision T3500 and Dell precision T3610) hooked up to a 19-in. LCD monitor (both 1280 × 1024 pixels, 49.9° × 39.9° of visual angle) at 60 Hz and an EyeLink 1000 kit (http://www.sr-research.com/). Eye-tracking data was collected at 250 Hz. Participants sat 57 cm from the monitor, their head stabilized with a chin rest, forehead rest, and headband. A protective opaque white screen encased the monitor and part of the participant's head in order to shield the participant from environmental distractions. 

Procedure

The study took place at the Live Science Pod in the Who Am I? exhibition of the London Science Museum. The room had no windows, and the ambient luminance was very stable across the experiment. It consisted of three phases for a total duration of approximately 15 min. Phase 1 was a 10-item personality questionnaire based on the Big Five personality inventory (Rammstedt & John, 2007), collected on a dedicated set of computers. Each personality trait (extraversion, conscientiousness, neuroticism, openness, and agreeableness) was assessed through two items. Item order was randomized across participants. Phase 2 was the eye-tracking experiment. Experimental design features, such as the task at hand or the initial gaze position, have been shown to drastically impact face exploration (Armann & Bülthoff, 2009; Arizpe, Kravitz, Yovel, & Baker, 2012). Here, they were standardized for every trial and participant. Participants were asked to freely look at 40 videos of one randomly selected actor. Prior to every trial, participants looked at a black central fixation cross presented on an average gray background. The fixation cross disappeared before the onset of the stimulus. In each trial, one of 35 possible clips for that same actor was presented. Because there were 40 trials, some clips were presented twice. At the end of each trial, participants were instructed to indicate via a mouse button press whether the amount of time the actor was engaged in direct gaze felt uncomfortably short or uncomfortably long with respect to what would feel appropriate in a real interaction. Each experiment was preceded by a calibration procedure, during which participants focused their gaze on one of nine separate targets in a 3 × 3 grid that occupied the entire display. A drift correction was carried out between each video, and a new calibration procedure was performed if the drift error was above 0.5°. Phase 3 was an actor face-rating questionnaire (Todorov, Said, Engell, & Oosterhof, 2008), in which participants rated on a 1–7 scale the attractiveness, threat, dominance, and trustworthiness of the actor they saw during the eye-tracking experiment. 

Eye data processing

We parsed the (x, y) eye position signal into fixations and saccades with a custom algorithm (Nyström & Holmqvist, 2010). This algorithm relies on an adaptive velocity threshold, making the event detection less sensitive to variations in noise level. Hence, detection thresholds varied between trials. Absolute thresholds are used to discard obvious outliers. We used the ones provided by Nyström and Holmqvist (2010, table 2: max saccade velocity = 1000°/s; max saccade acceleration = 100,000°/s²; min saccade duration = 10 ms, min fixation duration = 40 ms). We provide an example of such parsing, along with the corresponding gaze x-coordinate in Figure S6. We also provide the distributions of fixation duration and saccade amplitude for all observers and all trials in Figure S7. Both position and pupil dilation data were further processed through a custom filtering algorithm that substituted signal losses with position/pupil data interpolated from data recorded prior to and following the loss of signal. For every period of lost data (no size restriction), we performed a linear interpolation of the eye position (x and y coordinates) and variation in pupil size, using 100 ms of signal before and after loss. We did not set a limit on displacement between preceding and succeeding samples. We discarded all points that fell outside the screen. In average, signal losses represented 2.45% of the trials. Based on this small number, it is unlikely that the interpolation significantly changed the data. Figure S8 shows a histogram of the percentage of data loss for all observers and trials. Pupil diameter was expressed on a trial-by-trial basis as a percentage change in diameter with respect to a baseline measure obtained in a 200-ms window preceding the start of each trial. Environmental luminance was constant throughout the whole experiment duration. We postprocessed the pupil signal to minimize artifacts caused by eye position. We removed the pupil foreshortening effect artifact (Jay, 1962; Spring & Stiles, 1948) from the pupil data by implementing a correction technique based on a geometric model that expresses the foreshortening of the pupil area as a function of the cosine of the angle between the eye-to-camera axis and the eye-to-stimulus axis (Hayes & Petrov, 2015). To estimate the variability of eye positions of a given observer, we used a dispersion metric (Coutrot, Guyader, Ionescu, & Caplier, 2012). If n eye positions were recorded from an observer, p = (x_i, y_i)_{i ∈ [1..n]}, the eye position coordinates, the intraobserver dispersion D is defined as follows:  

 

 

The dispersion is the mean Euclidian distance between the eye positions of the same observers for a given clip. Small dispersion values reflect clustered eye positions. 

Scanpath modeling

Hidden Markov models

To grasp the highly dynamic and individualistic components of gaze behavior, we model participant's scanpaths using hidden Markov models (HMMs; Chuk, Chan, & Hsiao, 2014). This method acknowledges that visual exploration is a process that unfolds in a particular sequence over time and cannot be summarized by mere average fixation counts or durations within predefined regions of interest (ROIs). We first represent the average scanpath of each participant as a Markov model. A Markov model is a memory-less stochastic model used for representing probability distributions over sequences of observations or states (S_t)_{t ∈ [1…T]}. It is composed of (a) priors (the probability distribution over the initial state) and (b) a state transition matrix, defining P(S_t|S_t–1), which encapsulates the probability of travel between states. States can denote processes (e.g., scanning, reading, deciding; Simola, Salojärvi, & Kojo, 2008), isolated targets (e.g., letter, line; Haji-Abolhassani & Clark, 2013), but here, in the specific context of face exploration modeling, each state represents gaze falling on a ROI of the face. The distribution of eye positions (emission density) within each ROI is modeled as a 2-D Gaussian distribution, N(m, σ), with m the center and σ the covariance matrix (the generalization of variance to 2-D) of the Gaussian. Modeling HMM states with Gaussians instead of more isolated targets allows us to relax the point of interest, taking into account phenomena such as the dissociation between the center of gaze and the covert focus of attention and the imprecision of the human oculomotor system and of the eye-tracker. For more details on eye movement Bayesian modeling, we refer the reader to Boccignone's (2015) thorough introduction. All the HMM parameters (priors, between-state transition probabilities, mean and covariance of the Gaussians) are directly learned from the eye data. They are obtained with the Baum–Welch algorithm, a special case of the expectation–maximization algorithm (Bishop, 2006; Rabiner, 1989). We set the prior means of the Gaussian emissions to be the actors' nasion and the prior covariance matrix to be isotropic with SD = 200 pixels (roughly the same size as the facial features). We trained one model per participant using their eye positions subsampled at 30 Hz (one value was drawn from the raw signal every cycle). We chose this frequency for two reasons: (a) It conveniently matched the frame rate, and (b) training HMM with time-sampled eye positions instead of successive fixations allowed us to capture fixation duration information in the transition probabilities; 30 Hz is a good trade-off between complexity and scanpath information. To avoid local maximum problems, we repeated the training three times and retained the best performing model based on the log likelihood. We set the number of states to three (imposing the number of states is mandatory in order to cluster the Markov models; Chuk et al., 2014). We also performed the analysis with two and four states, see Figures S1 and S2. With only two states, the models lacked spatial resolution, and with K = 4, one state is often redundant. For this reason, K = 3 was deemed the best compromise. To cluster Markov models, we used the variational hierarchical EM (VHEM) algorithm for hidden Markov models (Coviello, Chan, & Lanckriet, 2014). This algorithm clusters a given collection of Markov models into K groups of Markov models that are similar, according to their probability distributions, and characterizes each group by a “cluster centre,” i.e., a novel Markov model that is representative for the group. For more details, the interested reader is referred to the original publication. 

Figure 1

 

Scanpaths as Markov models. (a) Illustration of seven out of the 405 individual scanpaths modeled as Markov models with three states. Each colored area corresponds to a state or ROI. Transition matrices show the probabilities of staying and shifting. (b) Markov model averaged over the whole population with the VHEM algorithm. (c) Temporal evolution of the posterior probabilities of being in the states corresponding to fixating the left eye, right eye, and to the rest of the face. Error bars represent SEM. See also Figure S3.

One advantage of this method is that it is totally data-driven; we had no prior hypothesis concerning the states parameters. To estimate the global eye movement pattern across all participants, we averaged the individual Markov models with the VHEM algorithm for a HMM (Coviello et al., 2014). This algorithm clusters Markov models into N groups of similar models and characterizes each group by a “cluster centre,” i.e., a model representative of that group. Here, we simply set N = 1. In Figure 1a, we show a representative set of seven out of the 405 Markov models with three states we trained with participants' eye positions sampled at 30 Hz. One can see the great variety of exploration strategies, with ROIs either distributed across the two eyes, the nose, and the mouth or shared between the two eyes or even just focused on one eye. Yet when Markov models are averaged over participants, the resulting center model (Figure 1b) displays a very clear spatial distribution: two narrow Gaussians on each eye and a broader one for the rest of the face. According to the priors, the probabilities at time zero, one is more likely to begin exploring the face from the left eye (66%) than from the background (32%) or from the right eye (only 2%). Moreover, the transition matrix states that when one is looking at the left eye, one is more likely to stay in this state (94%, with a 30-Hz sampling rate) than when in the right eye region (91%) or in the rest of the face (back, 89%). These values are backed up by the temporal evolution of the posterior probability of each state (Figure 1c). We clearly see a very strong left-eye bias during the first 250 ms. This bias persists throughout but decreases over time. 

Clusters of gaze behavior

To test whether this general exploration strategy accounts for the whole population or if there are clusters of observers with different gaze behaviors, we again use the VHEM algorithm. We set N = 2 and obtain two distinct patterns (see Figure S3). To identify the variables leading to these different gaze strategies, we use multivariate analysis of variance (MANOVA). MANOVA seeks to determine whether multiple levels of independent variables, on their own or in combination, have an effect on the affiliation of participants to one group or the other. Here, the categorical variable is binary (group membership). We tested three groups of independent variables: personality traits, face ratings, and basic characteristics (nationality, age, gender of the observer [GO], gender of the corresponding actor [GA], and identity of the actor). We did not include actor's gaze up time (300–10,300 ms) in the MANOVA as previous analyses showed that this variable does not correlate with any gaze metric except pupil dilation (Binetti et al., 2016). This is coherent with Figure 1, which shows that gaze dynamics plateau 1 s after stimulus onset. Only the basic characteristics group led to a significant separation between the two clusters, F(1, 404) = 7.5, p = 0.01, with observer and actor gender having the highest coefficient absolute value (see Table S1). Face ratings and personality traits failed to account for the separation between the two clusters, respectively, F(1, 404) = 1.4, p = 0.58, and F(1, 404) = 2.0, p = 0.55. We also ran a MANOVA with the three groups of independent variables combined and obtained similar results. Other N values have been tested; i.e., we tried to separate the initial data set into three and four clusters of gaze behavior, but the MANOVA analysis was unable to find significant differences between identified groups. We also performed the same analysis (N = 2) for HMMs with two and four states and obtained similar results, see Table S1 for basic characteristics MANOVA coefficients. Hence, GA and GO are the most efficient variables to use to separate exploration strategies into two subgroups. In the following, we closely characterize these gender-induced gaze differences. 

Gender differences

To characterize gender differences in face exploration, we split our data set into four groups: male observers watching male actors (MM, n = 119), male observers watching female actors (MF, n = 84), female observers watching male actors (FM, n = 106), and female observers watching female actors (FF, n = 96). Figure 2 displays the eye position heat maps of four individuals, each of them being characteristic of one of these groups. We first compared the simplest eye movement parameters between these groups: saccade amplitudes, fixation durations, and intraparticipant dispersion (i.e., eye position variance of a participant within a trial). In the following, the statistical significance of the effects of actor and observer gender has been evaluated via one-way ANOVAs. Pair-wise comparisons have been explored via Tukey's post hoc comparisons. We find that for male observers, fixation durations are longer (Figure 3a), F(3, 401) = 10.1, p < 0.001; saccade amplitudes shorter (Figure 3b), F(3, 401) = 8.5, p < 0.001; and dispersion smaller (Figure 3c), F(3, 401) = 12.9, p < 0.001, than for female observers. Actor gender does not influence these results. Note that these results are mutually consistent: Shorter saccades and longer fixations logically lead to lower dispersion. Gender also impacts on the increase of pupil diameter: This value is greater in MF than in any other group (Figure 3d), F(3, 401) = 2.8, p = 0.03, consistent with the Belladonna effect (Rieger et al., 2015; Tombs & Silverman, 2004). We used the VHEM algorithm to obtain the center Markov model of these four groups (N = 1 within each group). In all four groups, the spatial distribution of states is similar to the one depicted in Figure 1b. We computed the posterior probabilities of each state as we did for the whole population (Figure S4). We find that during the first second of exploration, the left-eye bias is stronger in FF than in MM (Figure 3e), F(3, 401) = 3.1, p = 0.02, with no difference between the other comparisons. We also show that one is less likely to gaze at the eyes of a same-sex actor than of a different-sex actor (Figure 3f), F(3, 401) = 2.9, p = 0.03. 

Figure 2

 

Heat maps of eye positions of four representative individuals with light color indicating areas of intense focus. Left: male observers. Right: female observers. Top: male actors. Bottom: female actresses. One can see that females tend to explore faces more than males, who stay more within the eye region. Females watching females have the strongest left-eye bias. See also Figure S4.

Figure 3

 

Gender differences in gaze behavior. (a) Female observers make shorter fixations and (b) larger saccades. (c) Females are more scattered than male observers. (d) Increase in pupil diameter was expressed as a percentage change in diameter with respect to a baseline measure obtained in a 200-ms window preceding the start of each trial. Males watching females show a higher increase in pupil diameter than other pairings. (e) Females watching females (FF) have a stronger left-eye bias. (f) Males and females are both less likely to gaze at the eyes of a same-sex actor than of a different-sex actor. Error bars represent SEM.

Gaze-based classification

These patterns appear systematic and rich enough to differentiate both actor and observer gender. To test this, we gathered all the gender differences in gaze behavior mentioned so far to train a classifier to be able to predict the gender of a given observer and/or the gender of the observed face (Figure 4). We started from a set of 15 variables (D_i)_{i ∈ [1…N]}, with N the total number of participants. Vector D_i gathers for participant i the following information: Markov model parameters (spatial [x, y] coordinates of the states ranked by decreasing posterior probability value and posterior probability of the left eye, right eye, and rest of the face averaged over the first second of exploration), mean intraparticipant dispersion, mean saccade amplitude, mean fixation duration, mean pupil diameter, peak pupil diameter, and latency to peak. We then reduced the dimensionality of this set of variables (D_i)_{i ∈ [1…N]} by applying a MANOVA. We applied two different MANOVAs: one to optimize the separation between two classes (M observers vs. F observers) and the other to optimize the separation between four classes (MM vs. MF vs. FM vs. FF). The eigenvector coefficients corresponding to each variable are available in Tables S2 and S3. To infer the gender of an observer j, we used quadratic discriminant analysis (QDA). We followed a leave-one-out approach: At each iteration, one participant was taken to test and the classifier trained with all the others. For each participant j, we trained a QDA classifier with Display Formula , EV representing the first two eigenvectors from the first MANOVA. To infer the gender of both observer j and of the corresponding actor, we followed the same approach with the first two eigenvectors of the second MANOVA. Both classifiers performed highly above chance level (two-sided binomial test, p < 0.001).  

Figure 4

 

Gaze-based gender classification. Solely relying on eye data, a classifier based on dimensionality reduction and QDA achieves 51.2% of correct classification with four classes (MM, MF, FM, FF, chance level = 25%) and 74.3% with two classes (M or F observers, chance level = 50%). Both classifiers perform highly above chance (one-sided binomial test, p < 0.001). The initial data set consists of spatial coordinates of the states ranked by posterior probability value, associated posterior probability averaged over the first second of exploration, intraparticipant dispersion, mean saccade amplitude, mean fixation duration, mean dispersion, mean pupil diameter, peak pupil diameter, and latency to peak. See also Tables S2 and S3.

Such a classifier is able to correctly guess the gender of the observer 73.4% of the time (two classes, chance level = 50%). It correctly guesses the gender of both the observer and of the corresponding face 51.2% of the time (four classes, chance level = 25%). We obtained similar results with linear discriminant analysis: 54.3% correct classification with four classes, 73.4% correct classification with two classes. 

Discussion

Understanding the precise nature of face perception is challenging as the face comprises a high-dimensional, dynamic information space (Jack & Schyns, 2015). In this study, we used novel data-mining methods that encapsulate the highly dynamic and individualistic spatiotemporal nature of gaze. Although a few previous studies have used Markov-based analysis with eye-tracking data to identify fixations and saccades (Salvucci & Goldberg, 2000), to infer observers' tasks (Haji-Abolhassani & Clark, 2014; Simola et al., 2008), or to build visual saliency models (Zhong, Zhao, Zou, Wang, & Wang, 2014), only a small number of recent studies have applied these techniques to face exploration (Chuk et al., 2014; Kanan et al., 2015). This approach is particularly powerful as faces feature very clear and stable ROIs (eyes, mouth, nose), allowing meaningful comparisons of Markov model states across stimuli and observers. Here, for the first time, we propose to jointly use Bayesian (Markov model clustering) and frequentist (MANOVA) inferences to assess the influence of a large set of variables on face exploration strategies. We tested variables related to observers' psychological profile (personality traits) and how they perceived actors' face (face ratings) as well as basic demographics, such as age, nationality, and gender. We found both the gender of the observer and of the actor to be the most efficient variables to separate the different recorded exploration strategies into two homogeneous subgroups. This outcome cannot be explained by differences between stimuli because observers' gender is balanced between stimuli as shown in Figure S5. This is backed up by the very low MANOVA coefficient associated with the identity of the actor (7.9e-3), see Table S1. Our model-based results are supplemented with more classic eye movement parameters, such as fixation duration, saccade amplitude, intraobserver dispersion, and pupilometry. In the following, we discuss the different strategies followed by different gender groups as well as their implication in perception bias. 

Males are less exploratory than females

We present three complementary metrics indicating that female observers further explore the face they are presented with irrespective of the gender of the actor: Females make shorter fixations and larger saccades and their eye positions are more scattered over the actor's face. Previous studies have reported the same pattern even with very different stimuli or experimental designs (Mercer Moss, Baddeley, & Canagarajah, 2012; Shen & Itti, 2012; Vassallo, Cooper, & Douglas, 2009). For instance, in Shen and Itti (2012), eye movements were recorded while participants watched and listened to different speakers in various outdoor settings. The authors showed that women saccade more often away from the face of the speaker, especially to the body. This difference in gaze behavior has been linked to the greater accuracy of women in the decoding of nonverbal cues (Hall, 1984; Hall & Matsumoto, 2004; McClure, 2000; Schmid, Mast, Bombari, & Mast, 2011). Actively looking for nonverbal cues distributed in many different parts of the face (Vatikiotis-Bateson & Kuratate, 2012) and body of the speakers, especially their hands (Krauss, Chen, & Chawla, 1996), would increase female's gaze dispersion. However, this hypothesis is undermined by the Reading the Mind in the Eyes test developed by Baron-Cohen, Wheelwright, Hill, Raste, and Plumb (2001), in which participants are asked to associate photographs of pairs of eyes with an adjective (e.g., playful, comforting, irritating, bored). Indeed, females are better than males at this task even though only the eye region is available (Kirkland, Peterson, Baker, Miller, & Pulos, 2013). Furthermore, males have been shown to be less exploratory than females even when exploring visual scenes without any face such as landscapes or art stimuli (Mercer Moss et al., 2012). 

Females looking at females have stronger left-side bias

We provide some insights into the time course of the global preference for the eye on the left side. We report a very strong left-eye bias during the first 250 ms of exploration, persisting throughout but decreasing over time. The left-eye bias is a well-documented, face-specific characteristic. This bias is very strong when exploring upright faces, is weaker in inverted faces, and disappears with nonface stimuli whether they are symmetric (vases, fractals) or not (landscapes) (Leonards & Scott-Samuel, 2005; Mertens, Siegmund, & Gruesser, 1993). It is often associated with chimeric faces: faces composed of two left halves are often judged to be more similar to the original face than faces composed of two right halves. Other studies have reported than when the left- and the right-hand sides of a face are different, observers tend to base their responses on the information contained in the left side. This includes face recognition (Brady, Campbell, & Flaherty, 2005); gender identification (Butler et al., 2005); and facial attractiveness, expression, and age (Burt & Perrett, 1997). The factors determining this lateralization remain unclear. Some obvious potential determining factors have been excluded, such as observers' eye or hand dominance (Leonards & Scott-Samuel, 2005). Other factors have been shown to interact with the left-side bias, such as scanning habits (left bias is weakened for native readers of right to left; Megreya & Havard, 2011) and start position (bias toward facial features furthest from the start position; Arizpe et al., 2012; Arizpe, Walsh, & Baker, 2015). The most common explanation found in the literature involves the right hemisphere dominance for face processing (Kanwisher, McDermott, & Chun, 1997; Yovel, Tambini, & Brandman, 2008). Although valid when fixation location is fixed at the center of the face, it may seem counterintuitive when participants are free to move their eyes (Everdell, Marsh, Yurick, Munhall, & Parè, 2007; Guo, Meints, Hall, Hall, & Mills, 2009; Hsiao & Cottrell, 2008). Indeed, looking at the left side of the stimulus places the majority of the actor's face in the observer's right visual field, i.e., their left hemisphere. An interpretation proposed by Butler et al. (2005) is that starting from a central position—either because of an initial central fixation cross or because of the well-known center bias (Tseng, Carmi, Cameron, Munoz, & Itti, 2009)—the left-hand side of the face activates right hemisphere face functions, making the latter initially more salient than its right counterpart. This interpretation is also supported by the fact that during a face identification task, the eye on the left side of the image becomes diagnostic before the one on the right (Rousselet, Ince, Rijsbergen, & Schyns, 2014; Vinette, Gosselin, & Schyns, 2004). This explains why the left-side bias is so strong during the very first moments of face exploration, but why does it persist over time? Different authors have reported a preferred landing position between the right eye (left visual hemifield) and the side of the nose during face recognition (Hsiao & Cottrell, 2008; Peterson & Eckstein, 2013). This places a region of dense information—the left eye and brow—within the foveal region, slightly displaced to the left visual hemifield, hence again activating right hemisphere face processing functions. An alternative hypothesis is that the left-side bias could be linked to the prevalence of right-eye dominance in humans. When engaged in mutual gaze, the dominant eye may provide the best cue to gaze direction through small vergence cues. Because the majority of the population is right-eye dominant (Coren, 1993), humans might prefer looking at the right eye (i.e., at the left side of the face) as it provides a clearer signal of mutual gaze. Here we found the left-side bias stronger in females looking at other females. This strengthening, coupled with the fact that the perception of facial information is biased toward the left side of face images complements an earlier study reporting that females are better at recognizing other female faces whereas there are no gender differences with regard to male faces (Rehnman & Herlitz, 2006). On the other hand, it is inconsistent with another study reporting that when looking at faces expressing a variety of emotions, men show asymmetric visual cortex activation patterns whereas women have more bilateral functioning (Proverbio, Brignone, Matarazzo, Del Zotto, & Zani, 2006). Further investigation is needed to disentangle the interaction between the gender of the observer and of the face observed in the activation of the right hemisphere face processing functions. 

Limitations of this study

The authors would like to make clear that this study does not demonstrate that gender is the variable that most influences gaze patterns during face exploration in general. Many aspects of the experimental design might have influenced the results presented in this paper. The actors we used were all Caucasian between 20 and 40 years old with a neutral expression and did not speak—all factors that could have influenced observers' strategies (Coutrot & Guyader, 2014; Schurgin et al., 2014; Wheeler et al., 2011). Even the initial gaze position has been shown to have a significant impact on the following scanpaths (Arizpe et al., 2012; Arizpe et al., 2015). In particular, the task given to the participants—rating the level of comfort they felt with the actor's duration of direct gaze—would certainly bias participants' attention toward actors' eyes. One of the first eye-tracking experiments in history suggested that gaze patterns are strongly modulated by different task demands (Yarbus, 1965). This result has since been replicated and extended: More recent studies showed that the task at hand can even be inferred using gaze-based classifiers (Boisvert & Bruce, 2016; Borji & Itti, 2014; Haji-Abolhassani & Clark, 2014; Kanan et al., 2015). Here, gender appears to be the variable that produces the strongest differences between participants. But one could legitimately hypothesize that if the task had been to determine the emotion displayed by the actors' face, the culture of the observer could have played a more important role as it has been shown that the way we perceive facial expression is not universal (Jack, Blais, Scheepers, Schyns, & Caldara, 2009). Considering the above, the key message of this paper is that our method allows capturing systematic differences between groups of observers in a data-driven fashion. 

Conclusion

Using the biggest and most diverse eye-tracking database recorded, we show that the way people look at faces contains systematic variations that are diagnostic of the gender of the observers and of the face they observe. These results have important implications in every field using gaze-based models. For instance, quantifying the nature of face processing and joint attention is critical to the understanding and the diagnosis of disorders, such as schizophrenia, autism, or attention deficit hyperactivity disorder (Freeth, Foulsham, & Kingstone, 2013; Wang et al., 2015). Tailoring these gaze-based models to a masculine or feminine population could lead to significant enhancements, particularly when a substantial sex ratio difference exists (e.g., autism). Going even further, one can speculate that different stimuli could elicit different systematic patterns diagnostic of other observers' characteristics, such as their state of health or level of cognitive development (Tseng et al., 2013; Wass & Smith, 2014). Given the ubiquitous nature of eye movements, being able to deduce such fundamental characteristics about a person without the need for self-report would have tremendous impact across a broad range of fields. 

Acknowledgments

Competing interests: We have no competing interests. Funding sources: A. C. has been funded by the Engineering and Physical Sciences Research Council (Grant No. EP/I017909/1). N. B., C. H., I. M., and A. J. have been funded by a Leverhulme trust grant (RPG-2013-218). Data accessibility: Eye data, stimuli, and demographics are available at http://antoinecoutrot.magix.net/public/databases.html. 

Commercial relationships: none. 

Corresponding author: Antoine Coutrot. 

Email: [e-mail]. 

Address: CoMPLEX, University College London, London, UK. 

Table S1

 

Coefficients of the first eigenvector of the MANOVA separating the two clusters of HMMs computed via the VHEM algorithm. Notes: We performed this analysis for HMMs with (a) 2 states, (b) 3 states, and (c) 4 states. Categorical variable: cluster 1 or 2. Independent variables: basic characteristics group. This group is the only one that led to a significant separation between the two clusters of Markov models. In the three analyses, the highest coefficient absolute values are the ones of Observer Gender and Actor Gender.

 

Table S2

 

Coefficients of the first and second eigenvectors of the MANOVA separating the eye data into four classes: MM, MF, FM, and FF. Notes: Categorical variables: gender of the observer and of the actor. Independent variables: state coordinates ranked by decreasing mean posterior probabilities; mean posterior probabilities of the left eye, right eye, and background; mean saccade amplitude; fixation duration and intraobserver dispersion; mean pupil diameter; maximum pupil diameter; and latency to maximum pupil diameter. The ratio of the between-group variance to the within-group variance for the first eigenvector is 0.39; 0.23 for the second.

 

Table S3

 

Coefficients of the first and second eigenvectors of the MANOVA separating the eye data into two classes: male and female observers. Notes: Categorical variables: gender of the observer. Independent variables: state coordinates ranked by decreasing mean posterior probabilities; mean posterior probabilities of the left eye, right eye, and background; mean saccade amplitude; fixation duration and intraobserver dispersion; mean pupil diameter; maximum pupil diameter; and latency to maximum pupil diameter. The ratio of the between-group variance to the within-group variance for the first eigenvector is 0.32; 3e-16 for the second.

 

Figure S1

 

Left: Illustration of five out of the 405 individual scanpaths modeled as Markov models with two states. Each colored area corresponds to a state or ROI. Right: Markov model averaged over the whole population with the VHEM algorithm.

Figure S2

 

Left: Illustration of five out of the 405 individual scanpaths modeled as Markov models with four states. Each colored area corresponds to a state or ROI. Right: Markov model averaged over the whole population with the VHEM algorithm.

Figure S3

 

Two clusters of gaze behavior. Centers of the two clusters of Markov models with three states obtained via the VHEM algorithm. Cluster 1 features narrow Gaussians centered on the eye region. Its priors are balanced between the three states. The transition probabilities are stronger for the eyes than for the rest of the face. Cluster 2 features broader states with priors favoring the left eye over the right eye. The transition probabilities are balanced between the three states.

Figure S4

 

Clusters of Markov model by observer and actor gender. Left: Markov models belonging to the MM, MF, FM, or FF group clustered via VHEM algorithm. Right: Corresponding posterior probabilities for the three possible states (left eye, right eye, and rest of the face). Error bars represent SEM.

Figure S5

 

For each actor, proportion of male and female participants. Actors 2, 3, 6, and 7 are male, the others are female. The analyses are performed on 405 participants (203 males, 202 females).

Figure S6

 

Example of an observer's gaze x-coordinate during a trial. Red vertical lines represent saccade onsets parsed with a version of the algorithm described in Nyström and Holmqvist (2010). This algorithm is able to cope with glissades, a type of wobbling movement that overshoots or undershoots the visual target. Previous studies showed the existence of small secondary saccades following a larger primary saccade (see, for instance, Wu, Kwon, & Kowler 2010). If such saccades coexisted in our data, our analysis does not allow separating them. However, reported differences in gaze behavior between men and women cannot only be due to such an oculomotor effect. Otherwise, we would not observe all the other differences in gaze dispersion, left-side bias, or pupil diameter. Moreover, the HMM modeling that we describe outputs Gaussian ROIs big enough to include both primary and the corresponding secondary saccades within the same ROI. The fact that we are able to classify men versus women mostly based on the parameters of such HMMs (cf. MANOVA eigenvector coefficients in Tables S2 and S3) shows that gender differences cannot be (only) explained based on primary and secondary saccade planning.

Figure S7

 

Distribution of fixation duration and saccade amplitude for all observers and all trials. We used 1,000 bins for both histograms.

Figure S8

 

Histogram of the percentage of data loss for all observers and all trials. We used 50 bins.