Modeling facial perception in group context from a serial perception perspective

Abstract

By utilizing statistical properties and summary statistics, the visual system can efficiently integrate perception of spatially and temporally adjacent stimuli into perception of a given target. For instance, perception of a target face can either be biased positively toward previous faces (e.g. the serial dependence effect) or be biased negatively by surrounding faces in the same trial/space (e.g. spatial ensemble averaging). However, both aspects were investigated separately. As spatial and temporal processing share the same purpose to reduce redundancy in visual processing, if one statistical processing occurs, would the statistical processing in the other domain still exist or be discarded? We investigated this question by exploring whether serial dependence of face perception (of attractiveness and averageness) survives when the changed face perception in the group context occurs. The results of Markov Chain modeling and conventional methods suggested that serial dependence (the temporal aspect) co-occurs with changed face perception in the group context (the spatial aspect). We also utilized the Hidden Markov modeling, as a new mathematical method, to model statistical processing from both domains. The results confirmed the co-occurrence of temporal effect and changed face perception in the group context for both attractiveness and averageness, suggesting potentially different spatial and temporal compression mechanisms in high-level vision. Further modeling and cluster analysis further revealed that the detailed computation of spatially and temporally adjacent faces in the attractiveness and averageness processing were similar yet different among different individuals. This work builds a bridge to understanding mathematical principles underlying changed face perception in the group context from the serial perspective.

Introduction

The human visual system has a remarkable ability to integrate information from both spatial and temporal environment (e.g. Webster, 2015). For instance, when we are communicating with a group of people, we capture their overall expressions and emotions at the same time (spatially) and process continuity of their expressions across time (temporally). In another case, as we are crossing a busy street, we need to form general perception of positions of people and cars (spatially) as well as effectively perceive their positions in a continuous sequence, especially motions and directions (temporally). Researchers currently who investigate visual perception no longer treat visual stimuli as totally independent but view them as dependent on each other. Based on cumulative evidence, several processing mechanisms have been proposed to account for visual perception of temporally and spatially adjacent stimuli. The human visual system is able to process an overwhelming amount of spatial and temporal information in a statistical manner. Roughly 100 years ago, Gibson (1933) already found that prolonged exposure to an adapting stimulus biased perception of following stimuli away from the property of the adaptor: the adaptation aftereffect (Webster, 2014; Webster, 2015). 

The temporally and spatially adjacent stimuli may impact perception in totally different ways (Ying, Burns, Choo, & Xu, 2020). In the spatial domain, ensemble coding, where human extracts a “gist” of a group with high accuracy, has been proposed to explain spatial and sequential visual perception (Whitney & Yamanashi Leib, 2017; Ying et al., 2020). Some recent studies suggested that face perception can be affected by the mean of the crowd (Whitney & Yamanashi Leib, 2017; Ying et al., 2020). To a certain extent, individuals may even mistakenly use the mean representation to represent an individual face (Haberman & Whitney, 2007). Moreover, more researchers found that a face would be perceived differently when presented in a group context from being viewed alone (Carragher, 2022; Carragher, Thomas, Gwinn, & Nicholls, 2019; Walker & Vul, 2014; Ying, Burns, Lin, & Xu, 2019). For instance, spatially adjacent attractive faces would make the target face being flanked to be less attractive or even unattractive (Burns, Yang, & Ying, 2021); spatially adjacent faces would possibly increase trustworthiness of a face (Carragher, Thomas, & Nicholls, 2021). Note that we tried to avoid using the old term the “cheerleader effect” to call these effects. Not only because it is borderline sexism, but also this old term (based on its initial definition and usage by Walker and Vul, who borrowed its name from a popular sitcom aired decades ago) may overly cover attractiveness change in female faces. The “Friend Effect” (used and advocated by Burns et al., 2021; Ying et al., 2019) could be a more proper term when discussing the “cheerleader effect,” as it is more appealing to use and covers perceptual change in the group context in facial traits other than attractiveness. 

On the temporal side, it has been found that the visual system uses serial dependence, a phenomenon where visual perception of the current stimuli is positively biased toward the recent past, to achieve temporal stability (Cicchini, Mikellidou, & Burr, 2018; Fischer & Whitney, 2014; Van der Burg, Rhodes & Alais, 2019). For instance, Xia et al. (2016) found that a temporally adjacent attractive face would alter perception of the current face to be more attractive. However, a temporally adjacent unattractive face would change perception of the current faces to be less attractive, which is characterized as serial dependence in facial attractiveness perception. More recently, Yu and Ying (2021) extended serial dependence to evaluation of various facial traits, such as dominance, attractiveness, and intelligence, supporting the ubiquity of serial dependence in face perception. 

However, some recent studies indicate that temporal and spatial statistical processing do not necessarily occur independently. Ensemble perception of current stimuli is biased positively toward the previous ensemble perception (Manassi et al., 2017), suggesting that spatial summary statistics are subject to temporal processing. Moreover, drawn on the ideal observer model, some new evidence highlighted the common function shared by spatial and temporal integration, which is efficient exploitation of redundancy of the rich and overwhelming visual environment (Cicchini et al., 2018, 2022). Note that, these findings were observed in low-level visual stimuli, we are still unclear how does the visual system utilize the spatial and temporal statistical information in high-level vision. 

If temporal statistical processing, which can reduce temporal redundancy, occurs, would the spatial statistical processing still exist or be discarded? Note that, although summary statistics would improve the signal-to-noise ratio of information (e.g. Alvarez, 2011), it will inevitably jeopardize the representation integrity of individual stimulus (e.g. Haberman & Whitney, 2007). Consequently, it might be efficient but also inaccurate to utilize summary statistics from both temporally and spatially adjacent stimuli to remove uncertainty and compress information. However, on the other hand, if statistical perception of the temporal and spatial information is processed for different purposes of information compression, they can co-occur without compromising the information integrity. Thus, studying the mechanisms behind the temporal and spatial statistical processing within the same paradigm would not only further unveil the complex characteristics of statistical processing but also clarify the shared and different mechanisms of information compression between temporal and spatial statistical processing. 

To answer these questions, we should be equipped with better modeling methods. Previous studies have established different methods to model serial dependence, the important mechanism of temporal processing. There are two popular methods to quantify serial dependence, one of which is absolute serial dependence (ASD). The ASD indicates how magnitude of the previous stimulus would influence current response errors of the current stimulus (e.g. fitting the data with the Derivative of Gaussian; Glasauer & Shi, 2022; Holland & Lockhead, 1968). The other common method is the relative serial dependence (RSD), which means the current response errors are dependent on the difference between the previous and the current stimuli (Cicchini et al., 2018; Glasauer & Shi, 2022). Taking these two approaches together, some researchers explained and modeled serial dependence mathematically, such as the Markov process (Yu & Ying, 2021) and the Kalman filter (which is based on the Markov property; Cicchini et al., 2018). The Markov process refers to a stochastic process in which the future state is dependent only on the current one but not the previous one (Häggström, 2002). In a recent study (Yu & Ying, 2021), authors found that (at least for) serial perception of facial characteristics follows a Markov property. This study suggested that future perception of a face is dependent on properties of the current face. This modeling attempt, together with other modeling studies in serial dependence (Kalman filtering; Cicchini et al., 2018), suggesting that it is better to understand sequential visual perception with mathematical modeling. 

However, it is hard to take spatial information into account using previous serial perception models. For example, when analyzing serial dependence using the Kalman-filter modeling, responses of the current stimuli were modeled as the weighted sum of the current and previous stimuli (Cicchini et al., 2018). Doing so, the spatial aspect would be largely ignored from the model. On the other hand, it is also hard to model this task with the Markov Chain directly either. Not every kind of sequential perception necessarily follows the Markov property: a specific facet of face perception is affected by spatially adjacent stimuli. In these studies (e.g. Burns et al., 2021; Ying et al., 2019), groups of faces were presented trial by trial, and participants were instructed to rate facial characteristics of the target face in the face group. Little did we know whether the changed face perception in the group context and serial dependence can occur simultaneously in an independent manner. As researchers have repeatedly found, the face judgment can either be biased by the face we just saw in the recent past (e.g. the serial dependence) or by the spatially adjacent faces in the context (e.g. the “friend effect”). Mathematical modeling which can capture both spatial and temporal input could serve to answer to the above-mentioned questions and help to shed light on the relationships of spatial and temporal statistical processing in high-level visual perception. 

What drives changed face perception in the group context? The hierarchical account (Walker & Vul, 2014) proposes that attractiveness judgment of the target face is biased positively by the surrounding faces as participants average the facial attractiveness toward the group means. However, previous studies also explained the cheerleader effect by the contrast effect in which averaged attractiveness of the surrounding flankers negatively predicts magnitude of the friend effect (Burns et al., 2021; Messner, Carnelli & Höhener, 2021; Ying et al., 2019). Thus, there are several mechanisms behind the friend effect: (1) the social positive effect, simply being surrounded by others makes a face more attractive as the target face is perceived as more popular among the group faces (Carragher et al., 2019; Ying et al., 2019); (2) the hierarchical encoding, the targets are averaged toward mean of the group via ensemble coding (Burns et al., 2021; Walker & Vul, 2014; Ying et al., 2020); and (3) the contrast effect, attractiveness of the surrounding flankers negatively predicts magnitude of the friend effect (Burns et al., 2021; Lei, He, Zhao, & Tian, 2020; Messner et al., 2021; Ying et al., 2019). The contrast effect may be the most important one because it can explain evidence of increased attractiveness (Carragher et al., 2019; Ying et al., 2019) and reduced attractiveness (the reverse friend effect; Burns et al., 2021) of the target face in the group context. Moreover, as suggested by the previous study (Ying et al., 2019), the contrast effect almost automatically occurs alongside (the attractiveness) ensemble coding of the surrounding faces. Specifically, Ying and colleagues (2019) suggested that mean attractiveness of the surrounding faces is extracted via ensemble coding and biases the target face judgment in a contrast manner. 

Note that the contrast between the group means and the targets is not directly observable to participants but still determines attractiveness perception. The friend effect is hard to be modeled with the Markov modeling, because the future perception is not driven entirely by the current stimuli but also by some not directly observable aspects of concurrent stimuli (e.g. the spatial effect of flankers). The Hidden Markov process can be an effective modeling method to take the underlying “hidden” factors in serial perception into account (Baum & Eagon, 1967). In a Hidden Markov process, the hidden states are not directly observable and can only be inferred from their relationship with the observed states (Barber, 2012). Moreover, the Hidden Markov models are able to unveil some “macroscopic structures” of the sequences. In a Hidden Markov model, the directly observed outputs (emissions) are called “observed states,” and the non-observable states of sequence that generated the observed outputs are called “hidden states” (Figure 1). In this study, to model the changed face perception in the group context across the time series, one may view the contrast effect between the surrounding faces and the target face as the “hidden states,” whereas the rating change of the target face as the “observed states.” The spatial and temporal mechanisms of face processing in the group context across the time series can be considered as the “macroscopic structure” of face processing within this Hidden Markov model. The probability density function that characterizes the “emission” from the hidden state (which is the spatial effect in this study) to the observed state (which refers to as the perceptual change of the target face here) is called the “emission probability.” The Hidden Markov modeling seeks to uncover the hidden states from the observed states. 

Figure 1.

 

An illustration of the Hidden Markov model. The observed data are denoted as X_i, whereas the hidden states (which drive the observed states) are denoted as Z_i. Here, we used Emission Probability to study the relationship between the contrast effect (hidden states) and the magnitude of the friend effect.

The Hidden Markov modeling can be an effective way to study serial perception in the group context and its mathematical property. To be noted, we also included judgments of face averageness in the analysis. The “averageness” is one of the facial traits, which indicates how typical the face is Vokey and Read (1992). Faces with greater averageness are more associated with ubiquitous structures of faces, whereas atypical faces which are with lower averageness are perceived as more distinctive or unusually seen in daily life (Vokey & Read, 1992). Following previous studies, we also asked the participants to rate “averageness” of the target face (Vokey & Read, 1992). Averageness is one of the determinant attributes of facial attractiveness, but it is not equal to attractiveness (DeBruine et al., 2007; Rhodes et al., 2001; Valentine et al., 2004). Analysis of facial averageness judgment serves as the cross-validation of our results in facial attractiveness perception. In addition, the previous study found that patterns of face perception change in the group context differ across different facial traits (Carragher et al., 2021). Therefore, exploring how perception of facial averageness changes in the group context would also shed light on how the effect of spatial context contributes to changed face perception in terms of different facial traits (Burn et al., 2021). Because facial attractiveness and averageness share the common mechanism but are still different facial traits, we expected to see common but different processing patterns of the friend effect. At first, we aimed to validate whether perception of facial attractiveness and averageness in a group context is affected by temporally adjacent trials via the Markov Chain modeling and Linear mixed-effect models (Glasauer & Shi, 2022; Yu & Ying, 2021). Then, we further studied the friend effect via Hidden Markov modeling as well as the Linear mixed-effect models and tried to better model the spatial and temporal aspects of face processing. We tested facial averageness (an essential but interdependent aspect of attractiveness; Little, Jones, & DeBruine, 2011) to further expand the scope of our modeling. To explore whether changed face perception in the group context is potentially different for different facial traits and whether such differences can be reflected in the Hidden Markov model, we applied the cluster analysis to the Hidden Markov models of the friend effect. As averageness is a main contributor to attractiveness, the common mechanism shared by the two facial traits should drive the common contrast effect of the friend effect in general. However, as averageness and attractiveness per se are different facial traits, the slightly different processing patterns of the contrast effect in the friend effect may be revealed by the cluster analysis. 

Methods

Participants

Twenty-three (23) consented volunteers with normal or corrected-to-normal vision participated (16 women and 7 men, mean age = 20.5 years). This study was approved by the Ethics Committee at Soochow University. We chose this sample size based on the power analysis (G*Power 3.1) of a recent study testing the friend effect using a similar paradigm on facial attractiveness and averageness (Burns et al., 2021). The results suggested we need at least 21 participants to reach a power of 0.95. 

Stimuli

To model the data more efficiently, we decided to test as many faces as possible. Therefore, current databases alone would not satisfy our needs. Although combining databases would be feasible, the lighting and quality difference may jeopardize the modeling. Therefore, we turned to other sources of facial stimuli. We used 150 computer-generated female faces based on a state-of-art computer vision framework (StyleGAN, a kind of Generative Adversarial Network; Karras, Laine, & Aila, 2021). In a typical GAN-framework image generating algorithm, two competing network models are trained one at a time iteratively: the Generator creates “fake” images, and the Discriminator estimates the probability that an incoming image comes from the training set rather than from the Generator. Competitions between the Generator and Discriminator drive the outcome of “fake” facial images indistinguishable from authenticated images. The stimuli were from a database which uses the StyleGAN framework (http://seeprettyface.com). Out of the 50,000 computer-generated faces from the database, we manually selected the 150 female faces (see a sample in Figure 2A) as stimuli used in the present study, which were: (1) Eastern Asian, (2) with a wide range of facial attractiveness, (3) with neutral facial expression, and (4) with frontal viewpoint and in a direct gaze direction. The faces were all presented in grayscale and were carefully aligned and trimmed via Photoshop (Adobe Inc.) at 200 × 200 pixels. 

Figure 2.

 

(A) A sample stimulus for the experiment. (B) The trial sequence of the experiment. The paradigm and spatial configuration of the surrounding faces and the central target face were adapted from the previous study (Ying et al., 2020). The face group was presented in a series.

Apparatus

The experiment was conducted at MATLAB R2016a (MathWorks) via Psychtoolbox (Brainard, 1997; Pelli, 1997) with a 27-inch LCD monitor (2560 × 1440 pixels, 120 Hz; see Zhang et al., 2018). During the test, participants sat with their chins resting on a chin rest placed at 53 cm away from the monitor (each pixel subtended 0.025 degrees). 

Procedure

The general procedure of this study is adapted from the previous study testing the friend effect (Ying et al., 2019). The experiment design is two (facial trait: attractiveness and averageness) by two (tasks: baseline [target face alone] and testing condition [with flankers]) within-subject experiment, resulting in four testing blocks for each participant. The orders of the four testing blocks were randomized across different participants. Specifically, for each facial trait, there is one block for the baseline condition and one block for the testing condition. To be noted, based on the previous study, “averageness” of the face, which was also referred to as “typicality” of the face, can also be rated directly (Vokey & Read, 1992). Therefore, we also asked participants to rate averageness of the target face, which indicates how typical the face is or the extent to which the face is associated with the ubiquitous facial structure (Vokey & Read, 1992). In the baseline condition, one of the 150 testing faces was randomly presented at the center of the screen alone (without a flanker). Participants were asked to rate attractiveness or averageness of the central faces in separate baseline blocks. In each baseline block, each of the 150 face stimuli was randomly presented twice, resulting in 300 trials (150 identities × 2 repetitions). The two ratings of each face stimulus in the baseline condition were averaged to obtain the baseline ratings of the 150 face stimuli. In the further analysis of the group condition, baseline ratings were used to calculate the average ratings of the surrounding faces which is critical to investigate the friend effect. The relatively precise estimation of the average ratings of the surrounding faces requires accurate baseline rating of each of the surrounding faces. Therefore, in the baseline condition, each face was rated twice to reduce other confounding factors (e.g. the order effect) in the baseline condition, which may potentially bias the baseline ratings. For each participant, the central testing faces in the testing condition were in the same order as the baseline blocks but were flanked by four surrounding faces. To randomly manipulate attractiveness or averageness of the flanking faces, the four flankers were randomly drawn from the 150 testing faces which have been rated individually in the baseline condition. The average ratings of attractiveness or averageness of the four flanking faces can be calculated by averaging their baseline ratings. 

At the beginning of each trial was a 1 second fixation: participants were asked to concentrate on the central fixational cross (Figure 2B). The testing face appeared on the screen for 1 second. After they disappeared, participants were asked to rate attractiveness of the central test face on a 1 to 7 scale (1 for the least attractive/averageness and 7 for the most attractive/averageness). The participants were asked to maintain their fixation at the center of the face (as much as possible) throughout the experiment. To guide fixations, the response screen had a fixational cross at the center (each line was 10 pixels × 2 pixels), and participants were asked to respond by pressing the corresponding keys on the keyboard. The fixational cross was presented at the screen. We acknowledge the fact that the fixation obscured a small proportion of the face. But note that the fixation covers only a tiny proportion of the nose. Moreover, the continuous presentation of the fixational cross, rather than a sudden removal of it at the “test faces” phase, would limit unforeseeable effects when perceiving the faces. On the other hand, some other studies in psychophysics have used the same or similar setting for attention modulation (e.g. testing ensemble coding: Ying et al., 2019; testing serial face perception, Yu & Ying, 2021). Therefore, even the face was slightly obscured by the fixational cross, we believe that the benefit outweighs the potential impact. 

Analysis

We analyzed the data via R version 3.4.3 (R Core Team, Vienna, Austria) and MATLAB R2018a (MathWorks) using different methods. The logic of the analysis follows: (1) using the conventional statistical methods to test existence of the changed face perception in the group context (the friend effect) and test the contrast account of it; (2) using the Markov Chain modeling and the linear mixed-effect model (as a classical approach to cross validate the Markov Chain modeling) to explore whether serial dependence exists in perception of a target face in the group context (and also whether follows the Markov property); (3) using the Hidden Markov modeling to qualify the contribution of the surrounding faces (the source of perceptual change of the target face in the group context) to the target face; (4) comparing the linear and nonlinear fitting of the emission probability matrix (see Figure 1) of Hidden Markov models to better describe the impact of the surrounding faces on the target face; and (5) using the K-means cluster analysis to further model the emission probability matrix of Hidden Markov models, and test whether the participants have one unified or several different pattern(s) of the contrast effect across time. 

First, we modeled the changed face perception in the group context using the conventional analysis (i.e. the repeated measures correlation and the linear mixed-effect model). Specifically, we analyzed how the target face ratings were influenced by the contrast between the average ratings of the surrounding faces and the target face ratings. 

To further investigate the potential serial effect between the target ratings in the group context. We used the Markov Chain modeling to examine the serial effect between the target face ratings (Yu & Ying, 2021). Specifically, each rating point (from 1 to 7) was considered as a rating state, and the Markov Chain modeling captures the transitional pattern among the rating states in a time series using the transitional probability matrices. The linear mixed-effect model was also used to validate the serial effect. 

In addition, as the serial effect was found in the target face rating (see Results section), the changed face perception in the group context was then investigated from the serial perception perspective. We used the Hidden Markov modeling (see Figure 2) to study the changed face perception in the group context. The hidden states represent the contrast effect (difference between the surrounding and the target faces), and the rating change of the target face (compared to baseline) can be considered as the observed states. In the Hidden Markov modeling, latent variables (Y₁, Y₂, ..., Y_N) are a Markov chain with values in the hidden state space {1, 2, ..., k}, where k represents the number of the hidden states (Pardoux, 2010). The Markov Chain of the latent variables is defined by two unknown parameters (μ, P), where μ represents initial state probability, and P is transitional probabilities between the latent variables (Pardoux, 2010). Under the condition of (Y₁, Y₂, ..., Y_N) = (y₁, y₂, ..., y_n), the observed sequence (X₁, ..., X_n) is independent. The distribution of each X_k is dependent only on y_k, and is the given function of yk. In other words, for any 1 ≤ n ≤ N,  

\begin{equation*}\begin{array}{@{}l@{}} P\left( {{X_1} = {x_1},{\rm{ }}...{\rm{ }},{X_n} = {x_n}|{Y_1} = {y_1},{\rm{ }}...{\rm{ }},{Y_n} = {y_n}} \right)\\ = \mathop \prod \limits_{k = 1}^n P\left( {{X_k} = \;{x_k}\backslash {Y_k} = {y_k}} \right)\;\\ = \;\mathop \prod \limits_{k = 1}^n {Q_{ykxk}} \end{array}\end{equation*}

 

Therefore, the probability of the observed sequence is defined as:  

\begin{eqnarray*} && {P_\theta } ( {Y_1} = {y_1}, {X_1} = {x_1}, {Y_2} = {y_2}, {X_2} = {x_2}, ...{\rm{ }}, \\ &&\quad {Y_N} = {y_N}, {X_N} = {x_N} ) \\ &&\quad = \mu \times P_{y1y2} \times {\rm{ }}...{\rm{ }} \times P_{yN - 1yN} \times Q_{y1x1}Q_{y2x2} \times \\ &&\quad {\rm{ }}...{\rm{ }} \times {Q_{yNxN}} \end{eqnarray*}

 

In the Hidden Markov modeling of this study, we set magnitude of the facial perception change in the group context as the observed states and the contrast effect between the group means and the targets as the hidden states. As a result, the responses in the serial perception of the surrounding faces can be modeled by the transition probabilities among the hidden states/factors and the emission probabilities from the hidden states/factors to the observed states/responses (see Figure 1). Participants’ successive responses to the targets are dependent on the trait values of the surrounding faces (spatial aspect). The changed face perception in the group context can be viewed as a Hidden Markov process. One object of the present study was to test whether the continuous visual perception can be understood as the Markov process by investigating the changed face perception in the group context from the serial perception perspective via the Hidden Markov modeling. 

To ease the operation of the Hidden Markov modeling, we qualified the hidden states and the observed states before investigation (see the Table 1). The contrast effect (the hidden states) was divided evenly into four levels by the quartiles, and perceptual change (the observed states) was also divided evenly into four levels by the quartiles. In Hidden Markov modeling, the transitional probability distribution among the hidden states is revealed by the transitional probability matrices. The emission probability matrices show the emission probability from the hidden states to the observed states. So, after determining the observed states and hidden states from the response sequences, we calculated the transition probability matrices among hidden states and the emission probability matrices from the hidden states to the observed states using the maximum likelihood estimate, which was achieved by the hmmestimate function from the Statistics and Machine Learning Toolbox of MATLAB. 

Table 1.

 

Division of the hidden states and the observed states for attractiveness and averageness. In addition, we fitted other kinds of matrices (Figure S1) which share a similar pattern which will be detailed in the “Results” section.

 

The cluster analysis was conducted for both attractiveness and averageness to further explore the between-individual variability in the contrast effect. Cluster analysis is an effective tool to divide similar data into clusters. As the emission matrices indicated how participants’ perception of the target faces changes (see Figure 2, emission states) as the function of the contrast between the surrounding faces and the targets (hidden layers), the difference in the emission matrices between participants can reveal the between-individual variability of the contrast effect. Therefore, after using the function to estimate the optimal number of clusters via the gap statistic (Tibshirani, Walther, & Hastie, 2001), we clustered the emission matrices via the K-means clustering process (Hartigan & Wong, 1978). 

Results

Summary of raw ratings under the baseline condition

The results showed high inter-rater reliability for attractiveness (Cronbach's alpha = 0.97) and averageness (Cronbach's alpha = 0.93). Participants used the full range of the rating scale (minimums = 1 and maximums = 7). The results showed that the rating distributions of attractiveness and those of averageness are similar (rho_Spearman = 0.86, p < 0.05). 

Analysis of the “Changed face perception in the group context” in the conventional way

The t-tests confirmed existence of the changed face perception in the group context at attractiveness (M = 0.21, SEM = 0.015, t(6899) = 14.44, p < 0.001, Cohen's d = 0.17), and averageness (M = 0.08, SEM = 0.018, t(6899) = 4.60, p < 0.001, Cohen's d = 0.06). The repeated measures correlation analysis (Figure 3; Bakdash & Marusich, 2017; Burns et al., 2021; Ying et al., 2019) suggested that the difference between the group mean and the target negatively predicts magnitude of the attractive (r = −0.44, p < 0.001, 95% confidence interval [CI] = −0.46 to −0.42) and averageness friend effect (r = −0.58, p < 0.001, 95% CI = −0.60 to −0.56). To cross-validate the results of the changed face perception in the group context, the linear mixed-effect model was used to investigate the influence of the contrast effect on the target face judgment in attractiveness and averageness, respectively. The results supported the contrast account of the changed face perception in the group context, which is consistent with the previous studies (Burns et al., 2021; Ying et al., 2019; see table S2 in the Supplementary Material). 

Validate the classic serial dependence of the target faces

Apart from using the linear mixed-effect model as a classic approach, we also used the Markov Chain modeling to capture how the rating states transit among the target faces from the serial perspective. In the Markov Chain modeling, each rating point (from 1 to 7) was considered as a rating state, and we can capture the transitional patterns between the rating states of the target face through the transitional probability matrices (Yu & Ying, 2021). We calculated the transitional probability matrices among target face ratings for attractiveness and averageness specifically (Figure 4), and then conducted matrix correlation analysis to test their correlation with the diagonal matrix, where values on the diagonal are consistently one and other values are zero. Such a diagonal matrix indicates the complete transition between the same rating states, which means the previous states are the same as the current ones. Matrix correlation analysis showed that the transition probability matrices are correlated significantly with the diagonal matrix for attractiveness and averageness (r_att = 0.50, p < 0.01; r_ave = 0.56, p < 0.01), indicating the positive serial effect of the previous target ratings on the current target ratings. To cross-validate the Markov Chain modeling, the linear mixed-effect model was also used to investigate serial dependence of facial attractiveness and averageness judgments, which supported the current results (see Table S1 in the Supplementary Material). 

Figure 3.

 

Transitional probability matrices in target face ratings (on the scale of 1–7) for attractiveness and averageness. For each figure, the color of each cell represents the size of transitional probabilities of each current response (y-axis) given a previous response (x-axis). Higher transitional probabilities are marked as red-er, whereas the lower transitional probabilities are marked as blue-er.

Analysis of “The changed face perception in the group context” from the serial perception perspective

Hidden Markov modeling

Two previous sections confirmed that (1) the changed face perception in the group context (and the contrastive effect) occurs via the repeated measures correlation and linear mixed-effect models; and (2) the facial attractiveness and averageness of a target face in the group context has serial dependence. It is reasonable to consider the temporal and spatial aspects of the changed face perception within the same model. We further studied the changed face perception in the group context using Hidden Markov models (considering the contrast as the hidden state) and plotted the emission probability matrices (Figures 5A, 5D). Both matrices clearly showed that the participants rated the target faces as more attractive/average when the surrounding faces were less attractive/average, which is indicated by the amber-colored cell emitted from the hidden-state-1 to the observed-state-4 from the matrices. However, when the surrounding faces were more attractive/average, participants tended to rate the target faces as less attractive/average (the navy-blue cell from the hidden-state-1 to the observed-state-1), and vice versa. Therefore, patterns of emission matrices supported the contrast effect underlying the friend effect. Further analysis showed that the emission probability matrices of attractiveness and averageness were similar (r = 0.80, p < 0.01), indicating the generally common processing pattern of the changed face perception in the group context shared by attractiveness and averageness. 

The results of the matrix correlation analysis indicated that the emission matrices of attractiveness and averageness are significantly different (both p values > 0.05) with a hypothetical matrix (all emission probabilities are equal to 1/4) without the contrast effect. However, they did not resemble the diagonal matrix (which represents a perfect contrast; attractiveness: r = 0.33, p = 0.21; averageness: r = 0.44, p = 0.09). These findings suggested that: (1) there exists the contrast effect in attractiveness and averageness, but (2) these contrast effect(s) are not simple linearly. 

Model fitting of the emission probability matrices of Hidden Markov model

Although previous studies tested the contrast effect using linear modeling methods (e.g. Burns et al., 2021; Ying et al., 2019); however, the shape of the Emission Probability matrices hinted that this may not be the case. Thus, we fitted the individual data with either a linear model or a Derivative of Gaussian (DoG; Yu & Ying, 2021). As we expected (see Figures 5B, E), the fitting results favored the DoG fitting for both facial traits (attractiveness: R²_DoG = 0.17 vs. R²_Linear = 1.55E-15; averageness: R²_DoG = 0.23 vs. R²_Linear = 1.06E-14). Thus, the contrast effect may have a nonlinear and negative effect on the changed face perception in group context for both facial traits at certain contrast levels. However, as predicted by the DoG models, the influence of the contrast effect on the face perception change tends to be attenuated at the extreme level of the contrast effect. 

Cluster analysis of the emission probability matrices of Hidden Markov model

Finally, we conducted the K-means cluster analysis to test whether there is/are one unified or several different pattern(s) of the contrast effect of individual participants based on the emission probability matrices of the Hidden Markov model (Barber, 2012). Before the K-means cluster analysis, we estimated the optimal number of clusters using the gap statistic (Tibshirani et al., 2001). Gap statistic estimates the fair K number by calculating the gap statistics for each K (Tibshirani et al., 2001). The minimal optical K was decided when the gap statistic of K is larger than the difference between the gap statistics of K + 1 and its standard error, Gap(K) > Gap(K + 1) - SEM(K + 1). 

For attractiveness, the gap statistics for K = 3 fitted the best, thus the results favored there exists three patterns/clusters of the contrast effect (see Figure 5C). Most participants (at 65%; 15 out of 23) are sorted as pattern 3 (dotted green line): the contrast effect is prominent at the observed-state-3 (being slightly more attractive) for the hidden-state-1 and 2 (less attractive surrounding faces), but at the observed-state-1 (being severely less attractive) for the hidden-state-3 and 4 (more attractive surrounding faces). Fewer people are sorted as the patterns 1 and 2 (both at 18%; both 4 people), clearly different from the pattern 3. However, for averageness, the results suggested there exists only one pattern of the contrast effect in changed perception of facial averageness in the group context (see Figure 5F). The contrast effect is stronger at the hidden-state-1 and 4 and less obvious at the hidden-state-2 and 3. In general, the K-means cluster analyses suggested that the detailed mechanisms of the contrast effect(s) are similar yet different in the attractiveness and averageness friend effect, and the relationships between the contrast effect and the changed face perception in the group context should be nonlinear. 

Discussion

In this study, to investigate whether the temporal and spatial statistical processing can co-occur in high-level vision, we explored the changed face perception (of attractiveness and averageness) in the group context from the serial perspective. The results suggested that (1) changed face perception in the group context (spatial integration) is also subjective to serial dependence (temporal integration); thus, it is necessary to model face perception considering both the spatial and temporal effect from other faces. By setting the contrast effect between the surrounding faces and the targets as hidden states and magnitude of the changed face perception in the group context as observed states, data suggested that (1) there is a common statistical processing mechanism shared by attractiveness and averageness across the spatial and temporal domain; and (2) however, the detailed computation of spatially and temporally adjacent faces in perception of attractiveness and averageness was similar yet different among different individuals: there were three patterns of the contrast effect for attractiveness, but only one pattern for averageness. 

At first, we tested the serial (temporal) influence of the previous stimuli and the spatial influence of surrounding stimuli on the target face perception separately. We validated the changed face perception in the group context and the contrast account of the friend effect by the repeated measurements correlation and the linear mixed-effect model. Consistent with previous studies (Burns et al., 2021; Ying et al., 2019; and partly agreed with Carragher et al., 2019), the results suggested that, for both facial attractiveness and averageness, the mean representation of the surrounding stimuli negatively predicts the ratings of the target face. Therefore, the contrast effect between the target stimulus and the surrounding stimuli is a robust component of the facial perception change in the group context on both facial attractiveness and averageness. We then confirmed existence of serial dependence of face perception in the group context. Data from the Markov Chain modeling suggested that both transitional probability matrices of attractiveness and averageness ratings correlated significantly with the diagonal matrix, indicating the perfect and strong serial dependence. The results were cross-validated using the linear mixed-effect model (the classic method). To sum up, perception of the current target face was not only negatively changed by the surrounding faces (as observed in the friend effect) but also positively biased toward the previous face (as observed in typical serial dependence). Despite the fact that temporally and spatially adjacent stimuli may impact perception in totally different ways (Ying et al., 2020), the spatial statistical processing could co-occur with the temporal processing in high-level vision. 

The co-occurrence of changed face perception in the spatial and temporal statistical processing sheds light on how the visual system utilizes temporal and spatial statistical information. Even though spatial and temporal statistical processing share the common purpose which is taking advantage of redundancy to compress and ease information processing in low-level vision (Cicchini et al., 2018, 2022), our results suggest that spatial and temporal statistical processing did not preclude each other in high-level face perception. Perception of the target face is changed by the spatial (the friend effect) and temporal (serial dependence) statistical processing simultaneously. Such results are consistent with the previous study which supported different manners underlying spatial and temporal statistical processing (Ying et al., 2020) and in line with the fact that spatial summary statistics are subject to serial dependence (Manassi et al., 2017). We speculate that the independent co-occurrence of the spatial and temporal effect in facial trait perception suggests that the compression mechanisms of the temporal and spatial information are potentially different in high-level vision. The results highlight that future studies could further investigate the integration and interaction between spatial and temporal effect of visual processing. 

Here, two sets of analysis suggested that both temporal effect (serial effect among target face judgment) and spatial effect (the changed perception of a target face in the group context). However, previous studies analyzed the temporal (Cicchini et al., 2018; Fischer & Whitney, 2014; Van der Burg et al., 2019) and spatial continuity (Walker & Vul, 2014; Ying et al., 2019) independently. These practices ease the modeling; however, they may ignore complexity of the human perception. It is difficult to simultaneously capture temporal and spatial information within one model in behavioral study. For instance, serial dependence in facial trait judgment has been investigated using the Markov Chain modeling (Yu & Ying, 2021). Although the Markov Chain modeling is effective to investigate temporal effect by showing how the current state is influenced by the previous one, the model considers perceptual states as observable and dependent on information from previous states and thus is limited in capturing spatial information. 

In this study, we also used Hidden Markov modeling as a new method to revisit the effect considering the temporal processing. The Hidden Markov modeling can be an effective method to build the bridge between the serial/temporal effect of sequentially presented stimuli and the spatial effect of other stimuli in the same trial/space on perceptual change of the target face. The Hidden Markov models can potentially unveil some macroscopic structure of the sequences, which was the spatial and temporal mechanisms of face processing in the group context across the time series in the current study. As in the facial attractiveness perception, perception of the target face stimulus can either be influenced by previous faces in a time series (e.g. the adaptation effect, serial dependence) or by surrounding faces in the same trial/space (e.g. the friend effect). The emission probability matrix captured by the Hidden Markov Modeling can represent the probability of rating change with different levels of the contrast effect between the target and the surrounding. Compared with the conventional analysis (e.g. repeated measures correlation and linear mixed-effect model) which have been used to investigate the changed face perception in the group context (as a linear model; Burns et al., 2021; Ying et al., 2019), the Hidden Markov Modeling is more efficient. This method not only represents how the rating change as a function of the contrast effect between the target face and the surrounding faces but also shed light on the detailed relationships between the rating change and the contrast levels, which is shown by the pattern of the emission probability matrices. Therefore, in the present study, we proposed the Hidden Markov model to model the contribution of both the spatial and temporal effect to changed face perception in the group context across time. 

Previous studies have proposed several hypothesized components of the friend effect (mostly on the facial attractiveness), including the social positive effect, the hierarchical encoding, and the contrast effect (Burns et al., 2021). These studies using conventional analysis methods have observed some contradictory results. For instance, the hierarchical encoding account (Walker & Vul, 2014) proposes that the target face is perceived as more attractive when the surrounding faces are attractive, as participants average the facial attractiveness toward the group means. However, Ying et al., (2019) first offered the contrast account, suggesting that the less attractive face group leads the target face to be more attractive due to comparison, which argued against the hierarchical encoding account. There were also other studies which provided evidence against the hierarchical encoding account. For example, morphed faces have been applied to the friend effect (Carragher et al., 2019). The study found that perceptual change of the target face failed to associate with attractiveness of the averaged morph face of the group (Carragher et al., 2019), which violated the hypothesis of the hierarchical encoding account. They also found that changed face perception of the target can occur even when the target face is surrounded by non-face stimuli (e.g. houses), suggests that the friend effect is not caused by the single hierarchical encoding mechanism that is specific to face processing (Carragher et al., 2019). Following their earlier work (Ying et al., 2019), those researchers found new evidence to reconcile the hierarchical encoding account and the contrast account within the friend effect framework (Burns et al., 2021). The friend effect framework proposed that highly attractive surrounding faces could conversely make the target face appears less attractive, suggesting the contrast account. However, the perceptual change in the group context is also related to the attractiveness level of the target, in which the more unattractive target benefits more in an unattractive group than the attractive target face, which to some extent supported the hierarchical account. Our current findings would explain these seemingly contradictory findings together. 

With the tentative DoG modeling (Fritsche, Spaak, & de Lange, 2020; Yu & Ying, 2021), we found the relationship between the contrast effect and the changed face perception in the group context is nonlinear. The nonlinear relationships between perceptual change of the target face in the group context and the contrast effect can explain the aforementioned accounts in a unified theory. Although the contrast effect could drive the perceptual change of the target face when the contrast effect is at the moderate level, the averaging process between the target and the group face may occur when the attractiveness difference between the target and the surrounding faces is at the extremes, which is in line with the assumption of the hierarchical encoding account. In other words, the greater benefit of the more unattractive target in the group context can be better understood as the nonlinearity of the friend effect which is confirmed by the present study. Future studies should further investigate this hypothesis. To be noted, it does not necessarily mean that the previous understanding of the contrast effect is wrong: with specific experimental designs (i.e. Ying et al., 2019; using 3 levels of the friend effect), the (non)linearity is differentiable. Indeed, as we used the computer-generated stimuli, Carragher and his colleague's study (2019) used computer generated morphed faces. Different stimulus types may have slightly different effects on the perceptual change in the group context. Future study can investigate this question using various types of face stimuli and other computational model and algorithm. 

On the other hand, the Hidden Markov model effectively investigates individual differences among participants. At least for studies focused on face perception in the group context, individual differences were primarily ignored. We generated individual Hidden Markov models and further analyzed individual differences by applying the K-means cluster analysis to the Hidden Markov models (Hartigan & Wong, 1979). Although the previous study which investigated face perception in the group context using the linear mixed model did not find a significant difference between facial attractiveness and averageness, the cluster analysis on the Hidden Markov models for attractiveness and averageness suggests that the detailed processing mechanisms of the changed face perception in the group context are slightly different. Specifically, the K-means cluster analysis favored one consistent pattern of emission matrices in averageness (see Figure 5F) but unveiled different patterns of emission matrices among participants for attractiveness (see Figure 5C). The cluster analysis applied here is an exploratory analysis of individual differences in face perception in the group context across time, which supports individual differences in processing mechanisms of face perception in the group context for attractiveness but not for averageness. Such a finding from Hidden Markov model follows the existing literature showing the complicated relationship between the two traits. While it is true that averageness (sometimes measured under the term of “distinctiveness”) is a vital part of attractiveness (e.g. Langlois, Roggman, & Musselman, 1994; Little et al., 2011; Perrett et al., 1999), it is well believed that perception of attractiveness is also formed by other physical (e.g. symmetry, skin color; and secondary sexual characteristics) and other factors (e.g. personality, Hormone level, rater's own attractiveness). Here in this study, the exploratory analysis using K-means cluster analysis differentiated the computational mechanisms of attractiveness and averageness: the relatively more complicated trait, attractiveness, can be computed via different mechanisms by different patterns; whereas the relatively less complicated trait, averageness, is unanimously computed. It is very likely that the tentative cluster analysis successfully captured the individual differences when computing these traits. For averageness, as its processing is primarily focused on the physical features of a face, different participants exhibited one pattern of the temporal and spatial integration. For attractiveness, as its processing is more complicated (individuals may incline to physical features disproportionally) and involves personal characteristics of the “rater,” the data suggested that there are (at least) three patterns of processing mechanisms. Therefore, cluster analysis in the Hidden Markov modeling supported that the detailed processing mechanisms of face perception in the group context might be different between attractiveness and averageness (see Figures 5C, F). Therefore, the Hidden Markov modeling has significant advantages over conventional methods. Future studies may employ different stimuli and modeling methods to further investigate the computational differences between these two traits. 

Figure 4.

 

Repeated measures correlation plots between the contrast effect and the magnitude of the “Changed Face Perception in Group Context” in (A) attractiveness and (B) averageness. Each unique color represents a single participant's data points and their trendlines (dashed lines).

Figure 5.

 

Emission probability matrices of attractiveness and averageness and the cluster analysis. (A), (B), and (C) are for attractiveness, whereas (D), (E), and (F) are for averageness. For A and D, the color of each cell represents the emission probabilities of the observed change in ratings of the target faces (y-axis) given the attractiveness levels of the surrounding faces (x-axis, hidden layer). Higher emission probabilities are marked as amber-er, while lower emission probabilities are marked as blue-er. For B and E, each participant's data were presented as a unique color, and the fitting curve represents the possible nonlinear relationship between the participant-wise averaged contrast effect (x-axis) and the changed face perception in the group context (y-axis). C and F showed the cluster analysis.

In this study, we are not suggesting that the Markov model is the only model that can explain the findings above, but it is a good method to model the serial and spatial effect on face perception. The Markov model has been used to investigate the transitional patterns and underlying functions of the serial effect. Specifically, the previous study (Yu & Ying, 2021) used the Markov Chain modeling to analyze serial dependence in facial trait judgment, and the results revealed a general mechanism of serial dependence facial trait judgment. The study applying Markov Chain modeling in face perception focused on how information from previous states influences current perceptual states, which is the serial effect from the temporal perspective (Yu & Ying, 2021). But sequential perception does not necessarily follow the Markov property: perception is affected by spatially adjacent stimuli. In other words, visual perception in real life is complicated, it not only integrates temporal information (Cicchini et al., 2018; Fischer & Whitney, 2014; Van der Burg et al., 2019) but also is dependent on information from surrounding stimuli at each time point (Ying et al., 2019). More often than not, the effect of spatially adjacent stimuli on the perceptual states cannot be observed directly, and cannot be simply captured by the Markov Chain model. Therefore, the Hidden Markov model serves as an extension of the Markov Chain model, as it can take spatial information into account by treating it as hidden states. Therefore, the Hidden Markov model builds a bridge between the serial/temporal effect and the spatial effect (Baum & Eagon, 1967) on visual perception. In addition, the Hidden Markov models are believed to be able to unveil the macroscopic structure of the sequences. By applying the Hidden Markov modeling to the face perception from the serial perspective, our results indicate that the Hidden Markov model can not only validates results from conventional methods but also reveals individual difference and nonlinearity of the face perception in group context with the emission probability matrix. We believe that the Hidden Markov modeling and other similar methods using Bayesian statistics (Fritsche et al., 2020), could and should unveil the complex mechanism within many effects involving temporal and spatial perception. Future studies should consider using more mathematical methods and comparing them. 

In conclusion, using the Markov Chain and the linear mixed-effect model, we found that changed perception of facial attractiveness and averageness in the group context also shows serial dependency, which suggests that the spatial and temporal statistical processing could co-occur in high-level vision. The results suggested different cognitive purposes of statistical perception of the temporal and spatial information. Moreover, alongside the analysis methods studying spatial and temporal aspects of face perception in the group context, we used the Hidden Markov modeling to analyze spatial and temporal aspects of face processing in the group context together. The emission probability matrices of Hidden Markov modeling provided further evidence that the changed face perception in group context is nonlinearly driven by the contrast between the ensemble representation of the surrounding flanking faces and the target. Moreover, using the K-means cluster analysis, data suggested the detailed computations of spatial and temporal adjacent faces in perception of attractiveness and averageness were similar yet different for different individuals. The findings expanded our understanding of serial face perception in the group context and offered a new analysis method to unveil the mathematical principles of the spatial relationship among stimuli from the serial perspective. 

Acknowledgments

The authors thank Yanqing Xu for the help of data collection. 

H. Ying is supported by the National Natural Science Foundation of China (32200850), Natural Science Foundation of Jiangsu Province (BK20200867), and the Entrepreneurship and Innovation Plan of Jiangsu Province. J.M. Yu conducted this study under the Undergraduate Research Advising Project. 

Data are available at: https://osf.io/h9dsj/. 

Authors’ contributions: J.M. Yu: data analysis, data visualization, and writing. W. Yang: coding and suggestion. H. Ying: conceptualization, coding and design, data analysis, data visualization, funding and resources acquisition, and writing. 

Commercial relationships: none. 

Corresponding author: Haojiang Ying. 

Email: [e-mail]. 

Address: Department of Psychology, School of Education, Soochow University, Suzhou, China.