Preferred fixation position and gaze location: Two factors modulating the composite face effect

Abstract

Humans consistently land their first saccade to a face at a preferred fixation location (PFL). Humans also typically process faces as wholes, as evidenced by perceptual effects such as the composite face effect (CFE). However, not known is whether an individual's tendency to process faces as wholes varies with their gaze patterns on the face. Here, we investigated variation of the CFE with the PFL. We compared the strength of the CFE for two groups of observers who were screened to have their PFLs either higher up, closer to the eyes, or lower on the face, closer to the tip of the nose. During the task, observers maintained their gaze at either their own group's mean PFL or at the other group's mean PFL. We found that the top half of the face elicits a stronger CFE than the bottom half. Further, the strength of the CFE was modulated by the distance of the PFL from the eyes, such that individuals with a PFL closer to the eyes had a stronger CFE than those with a PFL closer to the mouth. Finally, the top-half CFE for both upper-lookers and lower-lookers was abolished when they fixated at a non-preferred location on the face. Our findings show that the CFE relies on internal face representations shaped by the long-term use of a consistent oculomotor strategy to view faces.

Introduction

A quick look at a face allows us to glean a lot of socially relevant information, such as identity, gender, mood, attractiveness, and even trustworthiness. In the past half-century, there has been a lot of interest in understanding the mechanisms by which humans are able to achieve this feat (Bruce & Young, 2013). This interest in the mechanism of face processing has led to the discovery of many peculiar perceptual effects that occur in face judgments on stimuli that have been manipulated or doctored in ways that are uncommon or non-existent in nature. Notable examples of such effects are the face inversion effect (Yin, 1969), the Thatcher illusion (Thompson, 1980), the composite face effect (Young, Hellawell, & Hay, 1987), the parts versus wholes effect (Tanaka & Farah, 1993), and the face adaptation aftereffect (Webster, Kaping, Mizokami, & Duhamel, 2004). Although these effects are not ecologically valid in the sense that they do not have a functional role in real-life face recognition settings, their study has been central in our theoretical understanding of how humans process faces. Therefore, these visual tasks are an important testbed for studying face recognition. 

Recently, it was shown that across various common face tasks such as person, gender, and emotion recognition, humans consistently land their first saccade at a point just below the eyes (Peterson & Eckstein, 2012). This location, referred to here as the preferred fixation location (PFL), varies moderately across individuals (Peterson & Eckstein, 2013), and these variations generalize to the real world (Peterson, Lin, Zaun, & Kanwisher, 2016). This point of fixation plays a functional role in the aforementioned tasks, such that when observers fixate away from the PFL there is a reduction in task performance. More recently, it was shown that the internal representations of faces in humans are tuned to the PFL in these tasks (Tsank, 2019). Specifically, Tsank (2019) showed that individual differences in the variation of face identification performance with fixation position are best captured by an ideal observer model with internal templates as faces foveated at an individual's PFL. Further, the performance of convolutional neural network models trained on faces foveated at an individual's PFL on the face is also tuned to the PFL. These observations are consistent with research showing that human face processing is optimized for the face diet that we encounter during our lives (Oruc, Shafai, Murthy, Lages, & Ton, 2019; Yang, Shafai, & Oruc, 2014). This is because consistently moving one's eyes to a preferred location on the face can bias internal representations to match stimulus features accessible when fixating at the PFL, given peripheral information loss. This body of research has largely focused on the effect of the gaze position relative to the PFL on performance in common face tasks (identity, gender, and emotion identification), but we do not understand whether gaze position modulates the effect size in face tasks that are doctored to produce peculiar perceptual effects. As noted earlier, such research has the potential to elucidate constraints and mechanisms by which the long-term oculomotor strategy of consistently moving one's eyes to a PFL on the face can shape face perception. Here, we consider how the gaze position relative to the PFL modulates the composite face effect (CFE). 

The discovery of the CFE by Young et al. (1987) is an important landmark in the evolution of an influential hypothesis that upright faces are processed as units (Piepers & Robbins, 2012). In the CFE, the top (or bottom) halves of two faces, although identical, are perceived as being different when the bottom (or top) halves of the faces are different (Young et al., 1987). Over the years, the CFE has been studied extensively (Murphy, Gray, & Cook, 2017; Rossion, 2013). Figure 1 shows the general stimulus design for demonstrating the CFE. Note that the reader may not experience the CFE equally strongly for both halves. 

Figure 1.

 

The top-half and bottom-half CFEs. In a sequential face-half matching task, the irrelevant half affects the performance of the relevant half more when the faces are aligned as compared with when they are misaligned.

At its core, the CFE represents a surprising limitation of the visual system, which, despite its expertise in face processing, struggles to ignore irrelevant facial features, whereas it more effectively disregards irrelevant features of non-face objects (Cassia, Picozzi, Kuefner, Bricolo, & Turati, 2009; Robbins & McKone, 2007). Typically, this effect is attributed to holistic face-processing (i.e., a mode of computation that considers all the parts at once) (Richler, Palmeri, & Gauthier, 2012). There are several other examples of tasks where observers also show such inflexibility in adjusting for unfamiliar manipulations to features or viewing conditions. These include the inability to recognize inverted faces (Yin, 1969), diminished recognition performance with other-race faces (Cross, Cross, & Daly, 1971), atypical illumination directions(Braje, Kersten, Tarr, & Troje, 1998), diminished recognition for too small or too large faces (Yang et al., 2014), and diminished recognition when fixating away from the preferred first fixation location (Peterson & Eckstein, 2012). Thus, the CFE can be interpreted as an inability to flexibly use learned internal representations of full faces on a task that demands judgments based on partial use of incoming face information. In this general framework, we examined the role of long-term consistent eye movements to a consistent PFL on the face in shaping the internal representations of faces as measured by the CFE. 

For this, we implemented gaze-contingent versions of the classic CFE task, where we quasi-experimentally manipulated the PFL on the face and experimentally manipulated the half of the face being judged (top vs. bottom) and judgment fixation location (JFL) on the face. For this, we first screened observers whose PFL was either high up on the face close to the eyes (upper-lookers) or lower on the face close to the nose tip (lower-lookers). We then measured the strength of the CFE for the top and bottom halves of faces for the upper-lookers and lower-lookers when their JFL was either at their own group's mean PFL or at that of the other group's mean PFL. 

Our experimental design allowed us to test several hypotheses on how the CFE is modulated by variations in the PFL and JFL across the face. They are listed below: 

Hypothesis H1—Upper-lookers and lower-lookers would show a stronger CFE for the top half compared with the bottom half. This hypothesis is based on recent reports demonstrating the primacy of the eye region for face perception (Issa & DiCarlo, 2012; Rossion, 2013; Royer et al., 2018). 

Hypothesis H2—The amount by which the top-half CFE is stronger than the bottom-half CFE depends on the PFL, such that upper-lookers would show a greater top-half CFE than lower-lookers. This hypothesis is based on the idea that upper- and lower-lookers have different preferred internal representations, with the upper-lookers relying more on the eye region compared with the lower-lookers. 

Hypothesis H3—The CFE should be affected by the proximity of the JFL to the PFL, such that the CFE is stronger when the JFL is at the PFL and is diminished when the JFL is far away from the PFL. This hypothesis is based on the findings from Peterson and Eckstein 2012; Peterson and Eckstein 2013, showing that both upper- and lower-lookers perform best at face identification tasks when they fixate their PFL. 

The deidentified data, stimuli, and analysis scripts to reproduce the findings reported in this paper have been made publicly available via the Open Science Framework (OSF) and can be accessed at https://osf.io/vsdcj. 

Experiment 1

Methods

The study was conducted in three phases. First, we used a free-eye-movements face identification task to select a group of observers whose PFL was higher up on the face and another group whose PFL was lower on the face. We then conducted a free-eye-movements face identification task with these groups of observers on the four composite faces used in this study to verify that their PFLs are consistent with those measured in the prescreening task. After this, the observers completed a gaze-contingent sequential face-part matching task with the composite faces. In this task, the observers maintained their gaze at the average PFL of their own group or at that of the other group. This task was used to assess the strength of the CFE at the different gaze positions. 

Experiment design

We present data from two kinds of experimental paradigms in this paper: the free fixation face identification task and the enforced fixation sequential face-part matching task. The basic trial design for these two paradigms was the same across different experimental conditions. 

Free fixation face identification task

This task was used to measure an observer's preferred first fixation location on a face during face identification. To accurately measure the preferred first fixation location, each observer performed a total of 320 trials split across four blocks of 80 trials each. Within each block, the initial peripheral fixation location varied across eight peripheral locations around the screen. All participants completed the free fixation face identification task before we started running them on the enforced fixation task, which is described next. 

Enforced-fixation sequential face-part matching task

This task was used to measure the CFE for the top and bottom halves of faces while observers maintained their gaze at a specific location on the face (which was manipulated). We measured the CFE by comparing the performance on the matching task in the misaligned condition with that in the aligned condition. Each observer completed 2 (top-half/bottom-half) × 2 (aligned/misaligned) × 2 (fixation positions) × 256 (repeats) = 2048 trials. The trials were distributed into 16 blocks, with separate blocks for top-half/bottom-half judgments and aligned/misaligned conditions. Thus, there were four unique blocks, and each block was repeated four times. Within each block, there were 128 trials, and observer fixation was enforced in two different locations with equal probability. This task was completed in several sessions, usually over 7 to 10 days. Within each block, half of the trials were same trials, and the remaining half were different trials. Figure 4B shows the design of composite faces for these two trial types. Note that the stimuli for the same and different trials depend on whether the judgment is of the top half or of the bottom half. 

Participants

Prescreening task

A total of 126 observers participated in a short free-eye-movements face identification screening task. These participants were students at the University of California, Santa Barbara, who participated in the study for course credit or a small monetary reward. After consent for participation in the study and future contact were obtained, each participant was given instructions on how to perform the task. The experimental paradigm was identical to the standard free-eye-movements face identification task described in the Design section, except that it was a one-in-five face identification task using an in-house dataset of Caucasian faces. The dimensions and alignment of the Caucasian faces were matched to the composite faces. We measured the mean first fixation location on the face across trials when initiating a saccade from a peripheral location. 

Composite face tasks

To select upper-lookers and lower-lookers, we invited 15% of the participants of the prescreening experiment from the top and bottom ends of the distribution (about 19 participants from each group) for the full study. One upper-looker and two lower-lookers were unavailable to continue with the study. Thus, we had 17 upper-lookers and 16 lower-lookers. The upper-looker group consisted of 14 females and three males with ages ranging from 18 to 24 years. The lower-looker group consisted of 12 females and four males with ages ranging from 19 to 25 years. All participants were students at the University of California, Santa Barbara, and participated in the experiment in exchange for an hourly monetary compensation for a total of 7 to 10 hours. All participants had normal or corrected-to-normal vision. 

Stimuli

Prescreening task

We used a set of five frontally photographed Caucasian male faces for the prescreening task. These faces were a part of an in-house dataset. These faces were standardized by rotating, cropping, and resizing to align the eyes and the chin across the stimuli. We also matched the luminance and contrast energy to ensure that low-level features did not drive eye movements. The faces were about 12.6° × 9.2° in size. The images were presented at the full original contrast. Figure 2A shows model artificial intelligence (AI)-generated faces (using StyleGAN2) (Karras et al., 2020) that were processed similarly to visualize stimulus characteristics. The actual faces used in the study are not shown here. 

Figure 2.

 

(A) The left panel shows AI-generated model faces similar to those used in the one-in-five free fixation face-matching task used for prescreening (the actual faces are not shown to protect privacy). The right panel shows the four composite faces used in the main study (represented by AA, AB, BA, and BB). (B) Stimulus dimensions for the stimuli used in the prescreening task (left panel) and the composite face tasks.

Composite face tasks

We selected two Caucasian male faces photographed frontally for generating composite faces. These faces were part of an in-house dataset of faces. These faces were first standardized by rotating, cropping, and resizing them such that the eyes and chin were centered and aligned. These standardized faces were then converted to an 8-bit grayscale format and embedded in a mask that only revealed frontal facial features. Both the luminance and contrast energy were matched so that variations in skin color or texture could not be used to judge the identity. The faces were then split into two halves along the vertical dimension, and these halves were placed at a gap of 0.11°, as shown in Figure 4B. Different combinations of the top and bottom halves of these faces were assembled to create composite faces. They are referred to here as AA, AB, BA, and BB, with the first letter denoting the top half and the second letter denoting the bottom half (see Figure 2A). The faces were about 12.1° × 9.9° in size (see Figure 2B). Another corresponding set of four misaligned faces was formed by displacing the bottom half to the right by 2.4°. The stimuli were presented at 30% of the original image contrast to avoid ceiling effects in task performance. These settings are based on piloting to have an average identification performance of around 80%. Various other measurements between features in the face are shown in Figure 2B. 

Apparatus

The stimuli were presented on a Barco monitor (Barco, Inc., Duluth, GA) with a resolution of 1280 × 1024 and a refresh rate of 60 Hz. The monitor was calibrated linearly with a maximum luminance of 114.7 cd/m². The screen was placed 75 cm from the participant's eyes, such that each pixel on the monitor subtended a visual angle of 0.021° on their eyes. The stimulus display was controlled by software written using Psychtoolbox-3 (Brainard, 1997) running on MATLAB 2018 (MathWorks, Natick, MA). An EyeLink 1000 Plus desktop portable eye tracker (SR Research, Ottawa, ON, Canada) was used to track the left eye of each participant. The sampling rate of the tracker was 250 Hz. A nine-point calibration procedure was used at the start of each block and repeated periodically to ensure accurate gaze data recording. Standard EyeLink toolbox algorithms were used to identify saccade and fixation events from the gaze data. 

Trial design

Free fixation face identification

In this task, the initial fixation cross appeared in one of eight possible peripheral fixation locations (25° or 19° away from the center of the screen). Participants were instructed to maintain their gaze on the fixation location and hit a key to indicate readiness. After the keypress, the program checked in real time whether the participants maintained their fixation on the cross for a variable period of 500 to 1500 ms. If the observer's fixation drifted more than 1° from the center of the fixation cross, the trial was aborted and restarted. If the participant successfully maintained fixation for the variable delay period, the cross disappeared, and a noisy contrast-reduced face appeared at the center of the screen and stayed on for 1500 ms. This face was chosen randomly from the set of faces used for the experiment. Participants could freely move their eyes and examine the face during this period. After 1500 ms, the face disappeared and was replaced by a Gaussian white noise mask of matched mean luminance and a standard deviation of ∼6.4 cd/m² for 500 ms. This was followed by a response screen containing all faces from the dataset so the participant could select the face that was presented. The screen stayed on until the observer indicated which face they saw through a mouse click. After the response, the experiment progressed to the next trial. No feedback was given. See Figure 3A for a schematic of the events within a trial. 

Figure 3.

 

Schematics of the two tasks. (A) Free fixation face identification task: Observers initiated the trial by fixating one of the eight possible peripheral locations. A face was then presented in the center of the screen, and observers could freely move their eyes while studying the face. On the next screen, they were required to indicate which face was shown using a mouse click. (B) Enforced fixation sequential face-part matching task: On each trial, observers initiated the trial at one of the two possible fixation locations that differed by 2.1°. Then, two faces were flashed briefly, separated by a noise mask to wash out lingering percepts. While the faces were flashed, observers were prevented from drifting their gaze from the fixation location by more than 1°. After viewing the two faces, observers were required to respond to a question asking them to match a given half of the face (depending on the block). (C) Schematic of how the strength of the top-half and bottom-half CFEs were calculated. In each case, the percentage of correct responses in the aligned condition was subtracted from that in the misaligned condition to obtain the strength of the CFE.

Enforced-fixation sequential face-part matching task

In this task, the initial fixation cross appeared at two possible locations along the centerline of the face. The locations were chosen to be the average PFLs of the two groups (upper- and lower-lookers). As in the free-fixation face identification condition, observers were instructed to maintain their fixation on the initial fixation location and hit a key to indicate readiness at the beginning of the trial. After the keypress, the program verified if the observer maintained their fixation on the cross for a variable delay period of 500 to 1500 ms. The trial was aborted if there was an instance where the gaze drifted more than 1° from the center of the fixation cross. If the observer successfully maintained fixation on the cross through the delay period, a contrast-reduced composite face embedded in white noise with a standard deviation of ∼6.3 cd/m² was presented for 200 ms. Note that we added white Gaussian noise to the images during the presentation for modeling, the results of which are not discussed in this paper. After the presentation of the first face, a white Gaussian noise mask with mean luminance matched to the face and a standard deviation of ∼6.3 cd/m² was presented for 500 ms. This was followed by the second noisy, contrast-reduced composite face (chosen based on the current trial) for 200 ms. After this, a response prompt was shown that asked the observer to indicate with a keypress whether the half of the face being tested was the same or different between the two faces. This screen stayed on until a response was made. After the response, no feedback was given, and the next trial was initiated. In half of the blocks, the faces were misaligned. After the initial delay period, the fixation cross persisted throughout the trial (overlaid on the faces and mask) but with reduced contrast, helping the observer maintain fixation. The observer was instructed to maintain their gaze on the fixation cross throughout the stimulus presentation period. If their gaze drifted away by more than 1° from the fixation cross at any point during the trial, the trial was aborted and repeated. Figure 3B shows the trial schematic for this task. 

On each trial, the observer's response (same/different) was saved and compared with the ground truth to assess the percentage of correct responses for each condition. These accuracies were used to calculate the strength of the CFE for the top and bottom halves (see Figure 3C for a schematic and the Quantifying the CFE section below for further details). 

Procedure

The experiments were administered by trained graduate or undergraduate researchers in accordance with protocols approved by the institutional review board of the University of California, Santa Barbara. Participants were first briefed about the nature of the study and compensation agreements. After obtaining the subjects’ consent to participate in the study, they were given instructions about the task. The main experiment consisted of two tasks: the free-fixation face identification task and the enforced-fixation sequential part matching task. The two tasks were always performed in the same order; that is, the free fixation task was followed by the enforced fixation task. This was to verify that the participant had a consistent PFL for the composite faces that matched their PFL as measured in the prescreening task and to make the participants familiar with the four composite faces used in this experiment. Each task was further divided into blocks that took 15 to 20 minutes to complete. Participants were encouraged to take breaks between blocks and were not allowed to spend more than 1.5 hours per session to avoid the effects of fatigue. We recalibrated the eye tracker between blocks and whenever a participant took a break to maintain eye-tracking quality throughout each session. 

Analysis

Preferred fixation location

The preferred first fixation location is the first location inside the face that an observer's foveal gaze lands on when they move to a face from a peripheral fixation location. Following the completion of all blocks of the free fixation task for each observer, the preferred first fixation location was estimated as the mean fixation location across all of the first fixations on the face across trials. The vertical coordinates of the first fixation location on the face were used to analyze the dependence of CFE on the gaze position. 

Sequential part matching task performance

In this experiment, the observer was asked to report whether two sequentially presented composite faces contained the same or different portions of the part of the face they were instructed to focus on (either the top or bottom half). The performance in this task was calculated as the percentage of trials where the observer responded correctly. 

Quantifying the CFE

The strength of the CFE was measured as the difference in match task performance between the misaligned and the aligned version of the composite faces for a given condition (see Figure 3C for a schematic). Thus, the strength of the CFE is reported in percentage points. For example, if the strength of the CFE was 10%, the participant's performance in the misaligned condition was 10% higher than that in the aligned condition. Although this is the classic method to test the CFE, some recent studies (e.g., Richler, Gauthier, Wenger, & Palmeri, 2008) have shown that a signal-detection approach that yields sensitivity (d′) and bias (λ) metrics by considering the hits and false alarms in reporting a match on same and different trials is sometimes superior. We repeated our analyses using these metrics and found no qualitative differences. 

Power analysis

The choice of the number of subjects per group was made based on a priori power analysis done using data from a pilot experiment where we measured the effect of manipulating the enforced fixation position by 5.4° on the strength of the CFE. Effect sizes (Cohen's d) of 1.19 and 0.05 were obtained for the top- and bottom-half CFEs, respectively. It was thus reasonable for us to focus our power analysis on the top-half CFE. For a (1 – β) rate of 0.95, we would require a sample size of 12 (assuming a PFL manipulation of 5.4°). Thus, we would need a minimum of 12 observers per group (upper- and lower-looker) to have sufficient statistical power. Given the possibility that the difference in mean PFLs between the groups may not be 5.4° and the chance of participants dropping out, we aimed to prescreen enough participants to find 20 upper-lookers and 20 lower-lookers. 

We also conducted a power analysis to check how many participants would have to be screened to have a reasonable chance of finding two groups with 20 participants each that have the required distance between the mean vertical coordinates of the PFLs of each group. For this, we used an in-house database of PFLs measured on 186 participants. The y coordinates of the PFLs were normally distributed, as evidenced by a Kolmogorov–Smirnov test (p = 0.09). The sample mean of the y coordinates of the PFLs was 2.54° below the eye level with a standard deviation of 0.85°. Next, for different selected values of the mean difference between PFLs (ranging from 0.5° to 6°), we simulated samples of mean PFLs for different sample sizes drawn from the best-fit Gaussian distribution 10,000 times and counted the fraction of cases where the difference between the top and bottom 20 participants in the sample of PFLs was at least equal to the selected mean difference. This analysis resulted in an estimated probability of finding two groups of 20 participants, each with a required distance between PFLs and the number of participants to be prescreened (see Figure 4). We estimated that ∼120 participants would have to be screened for the distance of the PFLs between the two groups to be 2.1°. 

Figure 4.

 

This plot shows the results of the power analysis conducted using an in-house database of PFLs of 186 participants to estimate the number of participants required to be screened to find two groups of 20 observers that differ in their mean vertical coordinate of their PFLs by a given distance. The x-axis shows the expected number of observers to be screened. The y-axis shows the probability of finding samples with the mean difference indicated by the colormap shown to the right of the plot. The upper and lower dotted lines represent 80% and 1% chances, respectively. The chart suggests that we could expect to find two groups of 20 observers with their PFLs separated by about 2.1° with ∼80% chance if we screened about 120 participants.

Justification for the experiment design

If there is an effect of varying the PFL on the CFE, it would be easier to measure it if the distance in the mean PFLs across the groups is larger. However, because we are using a between-subjects design, controlling variations in extraneous low-level visual features between groups is essential. To keep the visual input consistent between the groups, we designed the experiment such that the PFL fixation for one group was the same as the non-PFL fixation for the other group. Thus, in this experiment, we aimed to maximize the distance between the PFLs of the upper-looking and lower-looker groups, given the constraint of finding enough participants from the tails of the PFL distribution as described in the power analysis above. 

Results

Selection of the upper- and lower-looker groups

The distribution of the vertical coordinates of the PFLs of the 126 observers in the prescreening task is shown in Figure 5A. A Kolmogorov–Smirnov test revealed that these values were normally distributed (p = 0.1). The distribution was slightly skewed toward locations lower on the face (sample skewness = 0.7). The upper- and lower-lookers were selected from the 15th-percentile tails of the distribution shown in pink and green, respectively. Thus, we selected about 18 participants from each group. This was done to maximize the chances of observing a significant effect of manipulating the PFL across groups. Some of the selected participants did not continue with the study, resulting in a final tally of 17 upper-lookers and 16 lower-lookers. 

Figure 5.

 

(A) The actual distribution of the vertical coordinates of the PFLs obtained from 126 screening participants. We selected the top and bottom 15% of the participants and invited them for further experiments. The upper-lookers are depicted in green, and the lower-lookers are depicted in pink. Those that were unavailable or not selected (due to poor-quality data) are depicted in brown. (B) Analysis of eye movements in the free-eye-movement face ID task using composite faces. The left half of the panel depicts the landing positions of the first eye movements of one example upper and one example lower-looker across 320 trials. Their PFLs are shown with green and pink crosses, respectively. The right half of the panel shows the PFLs of the upper- and lower-lookers that participated in this study in green and pink crosses, respectively. The mean PFLs of these groups are shown with a white circle and square, respectively. The fixation position while viewing the faces in the enforced fixation sequential face-part matching task was varied between these two spots across trials for both groups.

Verification of the PFL for upper- and lower-looker groups on composite faces

The first fixations on each trial of one upper-looker and one lower-looker are shown in the left panel of Figure 5B. The PFLs of the two groups are shown in the right panel of Figure 5B. The PFLs of the upper- and lower-looker groups were significantly different, t(31) = 8.41, p << 0.05. The average PFL for the upper-lookers was 0.43° below the eye level, and that of the lower-lookers was 2.45° below the eye level (see Figure 5). There was no significant difference in face identification performance across the two groups, t(31) = 0.22, p = 0.71, (percentage of correct) PC_{Upper-lookers} = 86.02%, PC_{Lower-lookers} = 85.17%. The two fixation positions for the enforced fixation task were chosen to be at the average PFLs of the two groups (see right panel of Figure 5B). 

The half being judged and the PFL modulate the CFE

The strength of the CFE was defined as the difference in face-part matching task performance between the misaligned and aligned conditions. To characterize the variation of CFE in our experiment, we conducted a three-way mixed-factor ANOVA with looker type (upper vs. lower), half being judged (top vs. bottom), and fixation position (average PFL of upper-lookers vs. that of the lower-lookers) as factors. There was a significant main effect of the half being judged, F(1, 30) = 22.72, p << 0.05, partial η² = 0.43, and a significant interaction effect of the half being judged and the looker type, F(1, 30) = 5.57, p = 0.025, partial η² = 0.16. The was also a significant interaction between the half being judged and the fixation position, F(1, 30) = 4.18, p = 0.049, partial η² = 0.12. No other main or interaction effects were statistically significant. The strengths of the CFE in various experimental conditions are plotted in Figure 6A. 

Figure 6.

 

(A) Results of the face-part matching task. The y-axis shows the strength of the CFE, which was calculated as the difference between the accuracy in the misaligned and aligned conditions. The box plot shows the strengths of the top and bottom CFEs for each participant at the two fixation locations. The filled and unfilled boxes represent the top-half and bottom-half CFEs, respectively. The vertical extent of the boxes represents the 95% confidence interval. Green and pink colors represent upper- and lower-lookers, respectively. Circles and squares represent conditions where observers fixated at the mean PFL of the upper-lookers and that of the lower-lookers, respectively. (B) The main effect of half being judged: The top-half CFE is significantly stronger than the bottom-half CFE. (C) Interaction effect of looker type and half being judged upper-lookers showed a significantly stronger top-half CFE than the lower-lookers. (D) Interaction effect of fixation location and half being judged: The top-half CFE was significantly stronger at the upper fixation position compared with that at the lower fixation position.

The main effect of the half being judged revealed that the CFE for the top half was significantly stronger than for the bottom half, irrespective of the looker type and the fixation position (µ_TopHalf = 9.7%, µ_BottomHalf = 3.0%) (see Figure 6B). Thus, our data confirm hypothesis H1 (higher top-half CFE than bottom-half CFE). The half being judged × looker type interaction effect revealed that the top-half CFE for upper-lookers was significantly stronger than that for lower-lookers (µ_{Upper-lookers} = 12.4%; µ_{Lower-lookers} = 6.6%; p = 0.0047, after adjusting for false discovery rate). This comparison was not significant for the bottom-half CFE (µ_{Upper-lookers} = 2.6%; µ_{Lower-lookers} = 3.3%; p = 0.61) (see Figure 6B). Thus, our data confirm hypothesis H2 (the preferred fixation location modulates the relative strength of the top-half CFE). We also had a weak half being judged × fixation location effect. Pairwise comparisons with false discovery rate corrections revealed that observers had a trend for a stronger top-half CFE at the upper fixation location compared with the lower fixation location (µ_TopFixation = 10.4%; µ_{LowerFixation} = 8.8%; p = 0.06). The strength of the bottom-half CFE was approximately the same at both fixation locations (µ_TopFixation = 2.7%; µ_{LowerFixation} = 3.3%; p = 0.45) (see Figure 6C). These data do not support hypothesis H3 (the CFE depends on the position of the judgment fixation location relative to the preferred fixation location). 

Discussion

We found that observers who have a PFL lower on the face (closer to the mouth) have a lower top-half CFE than observers with a PFL higher on the face (closer to the eyes), irrespective of their fixation position on the face. However, variations in the PFL did not modulate the bottom-half CFE. This may be caused by the low overall strength of the CFE (9.7% on average for the top half compared with 3.0% on average for the bottom half). The asymmetry in the strength of the CFE between the two halves agrees with previous findings (de Heering, Rossion, Turati, & Simion, 2008; Rossion, 2013; Young et al., 1987). Some researchers have noted that the reduced CFE for the bottom half can be remedied by matching the difficulty (of identification) of the two halves (Shyi & Wang, 2016). We did not match the difficulty of the two halves in our study. The bottom half judgments were significantly easier than the top half (see Supplementary Figure S1). The modulation of the CFE might have extended to the bottom half if we had matched the difficulty of the two halves. However, matching the difficulties using artificial means can make the faces look unnatural, which we wanted to avoid. 

Earlier research has shown that human performance on a variety of face tasks such as person, gender, and emotion identification is higher when the forced fixation location is closer to the PFL (Or, Peterson, & Eckstein, 2015; Peterson & Eckstein, 2012; Peterson & Eckstein, 2013). However, there are also face tasks such as ethnicity categorization where such a dependence of task performance on the proximity of the fixation point to the PFL is absent (Chakravarthula, Tsank, & Eckstein, 2021). We found no main effect of fixation location, which indicates that upper- and lower-lookers were not susceptible to fixation manipulation. Thus, these results might be interpreted that the CFE is unaffected by the fixation position; however, there is an important caveat to this result. The distance between the two fixation locations in our task design was 2.1° or 17.2% of the height of the full face. Peterson and Eckstein (2012) did not find a significant difference in accuracy on various face tasks when the observers were forced to fixate on the eyes versus the nose tip, which was 3°, or ∼20.5% of the height of the full face. They only found a significant degradation in performance at the mouth, which was 6° or 41% of the full-face height below the eyes. Thus, given the smaller distance across fixation locations for the manipulation in Experiment 1, the lack of a significant fixation effect does not rule out the possibility that the CFE depends on the distance of the fixation point from the PFL. To test whether larger manipulations of the fixation position on the face would result in a significant effect on the CFE, we conducted Experiment 2. 

Experiment 2

Methods

The primary aim of this experiment was to further evaluate the effect of JFL on the CFE (hypothesis H3) with a larger manipulation of the fixation position. For this, we needed to use a larger JFL manipulation of 5.4° or ∼44% of the vertical face dimension. In using the larger distance manipulation of 5.4°, we had to make another modification. For the upper-lookers, the non-preferred location was chosen to be lower on the face near the mouth/chin region. This could not be done for the lower-lookers, as the non-preferred location would be outside the face. Thus, we selected the non-preferred location higher up on the face (on the forehead) for lower-lookers. Because the comparison of interest was within subjects (across fixation positions on the face), the fact that the JFLs of upper- and lower-lookers were not matched in this experimental design is not a concern. This allowed us to keep the number of observers to be prescreened at approximately the same levels as in Experiment 1 while also allowing us to test a larger fixation manipulation of 5.4°. 

One may argue that measuring the CFE for one group (say, upper-lookers) at two different JFLs is sufficient to establish that the CFE depends on the proximity of the JFL to the PFL. However, such an experiment design would have a serious confound: If we find that the JFL modulates the CFE, then we cannot conclude if the effect was due to the change in position of the JFL relative to the face or the observer's PFL. For example, if the CFE of a group of upper-lookers is diminished when fixating the mouth relative to when fixating the eyes, is this due to looking away from their own PFL or is it due to looking away from the eyes? To address this issue, we again used two prescreened groups of upper- and lower-lookers. Now, if both upper- and lower-lookers showed a reduced CFE when their JFL was away from their PFL, then we could conclude that the proximity of the JFL to the PFL drove the reduction in CFE. On the other hand, if the upper-lookers but now lower-lookers showed a reduced CFE when their JFL was away from their PFL, then the reduction in the CFE was driven by the proximity to the eyes rather than proximity to the PFL. 

All other aspects of the experiment design, such as stimuli, apparatus, and trial design, were consistent with Experiment 1. As with Experiment 1, we used a prescreening task on an additional 132 participants to select upper- and lower-looker groups. Based on the measured PFLs, we tested 10 upper-lookers and nine lower-lookers. 

Nineteen undergraduate students at the University of California, Santa Barbara, participated in this experiment in exchange for course credit. Each participant first completed four blocks of the free-fixation face identification task depicted in Figure 3A. The eye movement data were used to locate the participant's PFL on the face. The participants then completed 16 blocks of the enforced-fixation sequential composite face-matching task depicted in Figure 3B. One enforced fixation location was chosen to be at the participant's own PFL along the midline of the face. The other enforced-fixation location was chosen 5.4° away (lower on the face for upper-lookers and higher on the face for lower-lookers), also on the midline of the face. We only allowed participants to proceed to the enforced fixation task if both the enforced fixation locations lay on the face. 

Results

The top-half CFE of both upper- and lower-looker groups is abolished when fixating away from their PFL

The PFLs of the 19 observers who completed both tasks as measured by the free-eye-movement composite face identification task are shown in Figure 7A. The average PFLs for the upper- and lower-looker groups were 0.63° and 2.44° below the eye level, respectively. The responses of participants on the matching task were analyzed using the same data processing pipeline as in Experiment 1 to yield the strengths of the top and bottom-half CFEs at the preferred and non-preferred locations for each participant. To characterize how the CFE varies with gaze position relative to the PFL, we conducted a three-way mixed ANOVA on the strength of the CFE with the half being judged (top vs. bottom) and the fixation position (PFL vs. non-PFL) as within-subject factors and the looker type (upper vs. lower) as a between-subjects factor. We found a significant main effect of the half being fixated, F(1, 16) = 8.11, p = 0.01, partial η² = 0.14, and of the fixation position, F(1, 16) = 11.2, p = 0.004, partial η² = 0.064. We also found a significant interaction effect between the fixation position and the half being judged, F(1, 16) = 15.63, p = 0.001, partial η² = 0.054. Planned comparisons revealed a significant difference in the top-half CFE when observers (both upper- and lower-lookers) fixated at their own-group PFLs versus when they fixated at a non-preferred location point 5.4° away on the face. The strength of the CFE for various conditions is visualized in Figure 7B. 

Figure 7.

 

(A) Mean first fixation locations of the 19 observers who participated in Experiment 2. (B) Two manipulated fixation locations used in Experiment 2 for each participant. One location was their PFL and the other was a non-preferred fixation located 5.4° away on the face. (C) The box plot shows the top and both half CFEs for upper- and lower-lookers in Experiment 2. The boxes indicate 95% confidence intervals.

Discussion

Experiment 2 showed that the top-half CFE is completely abolished when the fixation position is sufficiently distal to the PFL, irrespective of the preferred fixation location of the observer. This finding reconciles the apparent contradiction of the lack of a forced fixation effect on the CFE, considering the established result that human recognition abilities are tuned to individual PFLs (Peterson & Eckstein, 2012). The bottom half failed to elicit a significant CFE irrespective of the fixation position. This finding replicates our result from Experiment 1. 

It is important to note here that a significant impairment of the top-half CFE when fixating a non-preferred fixation location was seen for both upper- and lower-lookers. The non-preferred fixation locations of the lower-lookers were, on average, much closer to the eyes than those of the upper-lookers (2.8° vs. 6°, respectively). Thus, this finding precludes the possibility that the reduction of the top-half CFE was driven by the proximity of the gaze position to the eyes but was rather based on the distance of the JFL to the observer's PFL. 

General discussion

In this study, we examined how variations in where individuals typically focus (PFL) and gaze position affect the strength of the CFE. For this, we implemented a gaze-contingent sequential version of the classic CFE task, where observers maintained their gaze at a fixed location on the face while matching face halves. We quasi-experimentally manipulated the PFL by comparing groups of upper- and lower-lookers with PFLs near the eyes and nose tip, respectively. Our experimental design allowed us to study the interplay among three factors in driving the CFE: (a) predominance of face identity–related information in the top half of the face, (b) individual differences in internal face representations, and (c) variations in incoming face information due to changes in fixation position on the face. 

The distance of the PFL to the eyes modulates the CFE

Experiment 1 revealed that the CFE is stronger for the top half than for the bottom half for both upper- and lower-lookers. This result agrees with previous findings (de Heering et al., 2008; Rossion, 2013; Young et al., 1987) and is consistent with the primacy of the eye region for face recognition (Issa & DiCarlo, 2012; Royer et al., 2018). Critically, we also found a medium-sized effect of the PFL on the CFE, as upper-lookers showed a stronger top-half CFE compared with lower-lookers. No such effect was seen for the bottom-half CFE. 

Why might variations in the PFL modulate the CFE? We base our argument on three premises. First, there is convergent evidence that the internal representation of faces is tuned to the typical retinotopic position of facial features from perceptual experiments (Peterson & Eckstein, 2012; Tsank, 2019), functional magnetic resonance imaging (fMRI) experiments (de Haas, Schwarzkopf, Alvarez, Lawson, Henriksson, Kriegeskorte, & Rees, 2016), and electroencephalogram (EEG) studies (Stacchi, Ramon, Lao, & Caldara, 2019). Second, it has been suggested that visual experience shapes human face processing so that it is optimized for face attribute values typically encountered by the individual (Oruc et al., 2019; Yang et al., 2014). Third, the CFE is stronger when the faces are encountered in a familiar context. For example, the CFE is abolished for inverted faces (Young et al., 1987), reduced for other-race faces (Michel, Rossion, Han, Chung, & Caldara, 2006), modulated by the degree of visual experience (de Heering & Rossion, 2008), and stronger with faces viewed at a 2-meter distance compared with that at a 24-meter distance (Ross & Gauthier, 2014). We argue that the PFL forms a “familiar/learned context” for the observer. Individual differences in the PFL would result in different internal representations of faces. Thus, we suggest that modulation of the CFE by the PFL may be mediated by differences in prolonged visual experience across observers. Further, this visual experience may arise because of the long-term use of a consistent oculomotor strategy for viewing faces. 

Why might the upper-lookers show a stronger top-half CFE than lower-lookers? To understand this, we consider how the internal representations of faces that drive the CFE may differ between upper- and lower-lookers. The N170 component of the EEG response to faces has often been associated with many behavioral effects that are taken as evidence for holistic processing, such as Mooney face recognition (Eimer, Gosling, Nicholas, & Kiss, 2011; Latinus & Taylor, 2005), the composite face effect (Jacques & Rossion, 2009; Letourneau & Mitchell, 2008), the face inversion effect (Sadeh & Yovel, 2010). Further, the N170 component is also known to be strongly driven by the eyes compared with other parts (De Lissa, McArthur, Hawelka, Palermo, Mahajan, & Hutzler, 2014; Itier, Alain, Sedore, & McIntosh, 2007; Nemrodov, Anderson, Preston, & Itier, 2014). One possibility is that upper- and lower-lookers differ in the degree to which they use the information from the eye region for face processing, and this would manifest as differences in the N170 signals between these two groups. Indeed, in an abstract, Peterson, Or, Elliott, Giesbrecht, and Eckstein (2014) demonstrated preliminary evidence of an interaction effect between the PFL and the fixation location on the N170 signal. Thus, lower-lookers, who rely less on the top half of the face relative to upper-lookers, have a weaker CFE. Another possibility is that upper- and lower-lookers differ in the extent of face information they process. Chakravarthula and Eckstein (2024) recently demonstrated that upper-lookers show a more position-invariant face adaptation effect compared with lower-lookers, suggesting that upper-lookers represent a larger face region compared with lower-lookers. Thus, upper-lookers may be more affected by the information from the irrelevant half of the face. Based on these arguments, we suggest that the stronger top-half CFE for upper-lookers compared with lower-lookers may be mediated by the greater reliance on the eye region and a larger extent of face processing by the upper-lookers. 

Experiment 1 showed that upper-lookers have a stronger top-half CFE compared with lower-lookers. A double dissociation between the half being judged and the PFL on the face might be expected by finding that lower-lookers have a stronger bottom-half CFE compared with upper-lookers. However, lower-lookers did not have a significantly stronger bottom-half CFE compared with upper-lookers (there was a trend in this direction: µ_{Upper-lookers} = 2.6% and µ_{Lower-lookers} = 3.3% when averaged across fixation locations). One possible reason for this finding is that the manipulation of the PFL between the two groups (2.1° or 17.2% of the full-face height) was not large enough to capture an increase in the bottom-half CFE for the lower-lookers relative to the upper-lookers. Although all 17 upper-lookers had their PFLs located on the top half, six of the 16 lower-lookers had their PFLs on the top half. Why did we not choose a larger PFL manipulation across groups? Our choice of the PFL manipulation (2.1° or 17.2% of the full-face height) was based on a trade-off of the strength of the quasi-experimental manipulation of the PFL and the viability of the effort to find enough lower-lookers. This is further exacerbated by the lower baseline CFE magnitude for the bottom half compared with the top half (µ_TopHalf = 9.7%, µ_BottomHalf = 3.0%). With a lower bottom-half CFE magnitude, we will require an even larger sample size to detect a significant difference in the CFE between upper- and lower-lookers. Thus, it is likely that, even if we run the study with a sufficiently large sample of participants, the effect size of the difference between the bottom-half CFEs of upper- and lower-lookers will be much weaker than that for the top-half CFE. 

Looking away from the PFL diminishes the CFE

In Experiment 2, we used a larger fixation manipulation (5.4°, or ∼44% of the full-face height) while sacrificing the ability to match the visual input of the upper- and lower-looker groups at the two fixation locations. We chose upper- and lower-looker groups whose PFLs approximately matched those from the respective groups in Experiment 1. This time, observers either fixated at their own PFL or at a non-preferred location 5.4° (or 44% of the full-face height) away from the PFL. We found that the top-half CFE was abolished for both upper- and lower-lookers at the non-PFL, suggesting that the proximity of the fixation location to the PFL is an independent factor driving the top-half CFE. Critically, because the CFE was abolished for both groups, we can conclude that the CFE depends on the distance of the JFL to the PFL rather than a fixed facial feature such as the eyes. This finding is consistent with Peterson and Eckstein (2013), who demonstrated that individuals perform best at face identification when fixating their own PFL. The authors suggested that this pattern of results could emerge from using fixation-specific internal face representations and computations. Extending these arguments, we suggest that the magnitude of the CFE also relies on fixation-specific internal representations and computations. Specifically, if the incoming face information differs significantly from the typically accessed format at the preferred fixation location, the CFE will be diminished. 

Effects of the PFL on the CFE versus effects on signature tasks of holistic processing

Our results also suggest that there may be individual differences in the tendency to process faces as wholes (holistic processing), and these may be related to long-term differences in the PFL. However, it is important to note that our study only provides a partial picture of individual differences in holistic processing. Holistic processing has many meanings and measures (Richler et al., 2012). The CFE is one of the three gold-standard tasks alongside the face inversion effect (Yin, 1969) and the parts/wholes effect (Tanaka & Farah, 1993) that have traditionally been used to measure holistic processing. Recent results have shown that these measures only partially agree, suggesting that they may measure distinct aspects/perceptual mechanisms of holistic processing (Boutet, Nelson, Watier, Cousineau, Béland, & Collin, 2021). Therefore, our finding that upper- and lower-lookers showed a significant difference in the CFE may or may not translate to the other signature effects of holistic processing. Further experimentation is necessary to fully understand how changes in the PFL influence the face inversion effect and the parts/whole effect. 

Potential implications for understanding mechanisms underlying behavioral disorders

This paper used idiosyncratic variations in PFLs across the population to study how eye movements shape the CFE. Where might we find reliable differences in face-fixation behavior? One possibility is in populations with autism spectrum disorder (ASD). Tanaka and Sung (2016) discussed the eye avoidance hypothesis, where ASD individuals avoid looking at the eyes because they are perceived as socially threatening. Although ASD individuals do avoid prolonged fixation close to the eyes, Schauder, Park, Tsank, Eckstein, Tadin, and Bennetto (2019) showed that the initial eye movement to faces in ASD adolescents is similar to that of neurotypical adolescents. Further, no reliable evidence of impairment of holistic processing has been found in ASD individuals (Tanaka & Sung, 2016). Another possibility is populations with acquired prosopagnosia (AP). Caldara, Schyns, Mayer, Smith, Gosselin, and Rossion (2005) showed that the usage of eye information is strongly affected in AP patients. Their first eye movements to faces are predominantly to the mouth region (Orban de Xivry, Ramon, Lefèvre, & Rossion, 2008). Further, AP selectively impairs holistic perception (Ramon, Busigny, & Rossion, 2010) but not other judgments on faces (Jiang, Blanz, & Rossion, 2011; Quadflieg, Todorov, Laguesse, & Rossion, 2012; Van Belle, Busigny, Lefèvre, Joubert, Felician, Gentile, & Rossion, 2011). Interestingly, developmental prosopagnosia patients show similar face fixation behavior (Peterson, Hoke, Zaun, Duchaine, & Kanwisher, 2017) and CFE (Biotti, Wu, Yang, Jiahui, Duchaine, & Cook, 2017) as neurotypical subjects. Our finding that upper-lookers show a stronger CFE than lower-lookers provides further empirical evidence strengthening the connection between long-term oculomotor strategies and behavioral effects typically associated with holistic processing. Many other factors are known to bias face scanning to a mouth-focused pattern, such as bilingualism (Ayneto & Sebastian-Galles, 2017), 22q11.2 deletion (Campbell, McCabe, Leadbeater, Schall, Loughland, & Rich, 2010), and looking at faces with cleft lip and palate (Meyer-Marcotty, Gerdes, Reuther, Stellzig-Eisenhauer, & Alpers, 2010). Their effect on the CFE is currently unknown. 

In summary, our results demonstrate that the CFE is modulated by variations in both the gaze position and preferred fixation position on the face. These findings shed light on how the prolonged use of a consistent oculomotor strategy to view faces shapes internal face representations. 

Acknowledgments

Commercial relationships: none. 

Corresponding author: Puneeth N. Chakravarthula. 

Email: [email protected]. 

Address: Department of Psychological and Brain Science, University of California, Santa Barbara, CA, USA.