Holistic processing and visual characteristics of regulated and spontaneous expressions

Abstract

The rapid and efficient recognition of facial expressions is crucial for adaptive behaviors, and holistic processing is one of the critical processing methods to achieve this adaptation. Therefore, this study integrated the effects and attentional characteristics of the authenticity of facial expressions on holistic processing. The results show that both regulated and spontaneous expressions were processed holistically. However, the spontaneous expression details did not indicate typical holistic processing, with the congruency effect observed equally for aligned and misaligned conditions. No significant difference between the two expressions was observed in terms of reaction times and eye movement characteristics (i.e., total fixation duration, fixation counts, and first fixation duration). These findings suggest that holistic processing strategies differ between the two expressions. Nevertheless, the difference was not reflected in attentional engagement.

Introduction

Facial expressions display rich social signals that guide interpersonal communication, such as emotion and intention (Savage & Lipp, 2015). Based on the behavioral ecology view of facial displays, facial expressions as social tools serve as lead signs to contingent action in social interaction (Crivelli & Fridlund, 2018). That is, facial expressions are a means of achieving social influence, not merely representations of internal states (Parkinson, 2021). Therefore, facial expression perception is crucial for social interaction. 

However, people not only express genuine emotions but also regulate their facial expressions (Crivelli, Carrera, & Fernández-Dols, 2015; Yan, Wang, Zhang, Song, & Sun, 2016). In the interaction process, instead of reliable emotional states, facial expressions tend to display specific intentions deliberately (Jia, Wang, Hu, Webster, & Li, 2020). Therefore, understanding the authenticity of facial expressions has become an important yet challenging task in human behavioral research (Bartlett, Hager, Ekman, & Sejnowski, 1999; Jia et al., 2020). To the best of our knowledge, the research on processing the authenticity of expressions is limited. Hence, this study aimed to investigate the holistic processing of different authenticity expressions and their attentional characteristics. Fundamentally, it not only helps us understand the nature of the authenticity of expressions but also contributes to fresh insights into holistic processing. 

Expression processing strategies

Studies on facial expression indicate that individuals generally adopt a holistic processing strategy to evaluate facial expressions (Guo, 2012; Omigbodun & Cottrell, 2013). Although few researchers would disagree that the recognition of facial identity is holistic, there is less consensus on how the perception of different facial expressions is based on the holistic perception of the entire face (Tanaka, Kaiser, Butler, & Le Grand, 2012). 

Many researchers have observed a holistic processing of expression (Flack et al., 2015; Omigbodun & Cottrell, 2013). For example, Tanaka et al. (2012) found that facial expression processing is holistic when there is a clash between parts of a facial expression (e.g., angry–happy composite) but is analytic or parts based when there is little or no conflict between the parts (e.g., normal happy face). Holistic processing appears at a very early stage, as fast as 17 ms (Liu & Tanaka, 2019; Tanaka & Xu, 2018; Tanaka et al., 2012). Recently, researchers discovered evidence of viewpoint-invariant gaze patterns in facial expression appraisal, demonstrating that holistic mechanisms manifest during early perceptual stages (Gregory, Tanaka, & Liu, 2021; Guo & Shaw, 2015). Accordingly, the conflicting expressions may be more likely to be processed as holistic, and holistic processing of expressions perhaps occurs at an early stage. 

Processing of regulated and spontaneous expressions

During social interaction, not all facial expressions reflect actual emotions. Individuals can consciously regulate and suppress their emotions and express unfelt emotions according to their intention (Crivelli et al., 2015; Yan et al., 2016). For example, people typically laugh not only when they are genuinely happy but also when they hide sadness to maintain harmonious interpersonal relationships (Ambadar, Cohn, & Reed, 2009; Niedenthal, Mermillod, Maringer, & Hess, 2010). Thus, such behavior raises concerns about expression authenticity. 

Genuine expressions are coupled with emotional states. Be that as it may, regulated expressions are not coupled with the corresponding emotion and reflect the deception intent while ignoring the emotional state (Ekman & Rosenberg, 2005). Therefore, discerning the authenticity of expression is vital to ensure that subsequent behavior is appropriate (Zloteanu, Krumhuber, & Richardson, 2018). Misidentifying expressions could result in adverse outcomes. For example, approaching an individual who is smiling but is angry can lead to an avoidable confrontation (McLellan, Johnston, Dalrymple-Alford, & Porter, 2010). Indeed, some evidence has suggested some differences between regulated and spontaneous expressions in social perception. For example, it was observed that individuals prefer to trust and work with others who exhibit a spontaneous expression (Centorrino, Djemai, Hopfensitz, Milinski, & Seabright, 2015; Johnston, Miles, & Macrae, 2009). Therefore, authenticity may significantly contribute to face processing as an essential dimension of expressions. 

To shed light on this question, this study examined two types of expressions from the perspective of expressive authenticity—namely, regulated expression, which conveys the opposite of the original emotion, and spontaneous expression, which is a natural expression of emotion (Buck, Powers, & Hull, 2017). Previous studies have demonstrated that facial cues (e.g., race, sex, expression intensity) influence expression processing (Craig, Zhang, & Lipp, 2017; Murphy, Gray, & Cook, 2016). However, it is currently unclear how the expressions are processed to extract the authenticity information. Indubitably, it is necessary to understand the processing of regulated and spontaneous expressions to better understand the nature of expression processing. Hence, the present study used a holistic processing paradigm to explore holistic processing. 

Holistic processing is the ability or tendency to process overall facial features, which is usually evaluated using composite-face paradigms (Omigbodun & Cottrell, 2013; Richler & Gauthier, 2014). The composite-face task is either a partial design or a complete design (Gauthier & Bukach, 2007). Compared to the partial design, the complete design can eliminate the interference of response bias and has a better effect size (Richler & Gauthier, 2013; Richler & Gauthier, 2014). The complete design has been examined and validated extensively; therefore, this study used the complete composite-face task. In the complete design, the top and bottom face halves of a chimeric face could be either identical or different. If the top and bottom face halves are identical or entirely different, it is a congruent condition; otherwise, it is incongruent. 

Eye movements

Eye movements are functional in face processing (Altair, Tanaka, & Dawel, 2021; Henderson, Williams, & Falk, 2005; Schwarzer, Huber, & Dümmler, 2005; Tanaka, Liu, & Martin, 2021) and are often used to study the holistic processing of faces (Bombari, Mast, & Lobmaier, 2009; de Heering, Rossion, Turati, & Simion, 2008; van Belle, Lefèvre, Laguesse, Busigny, De Graef, Verfaillie, & Rossion, 2010). Specifically, the total fixation duration, fixation counts, and first fixation duration in eye movements are effective indicators of attentional engagement (Calvo, Gutiérrez-García, Avero, & Lundqvist, 2013; de Heering et al., 2008). Thus, this study used these three indicators to explore the attentional engagement differences between regulated and spontaneous expressions in holistic processing. 

Moreover, researchers have noted that humans can rapidly encode information from faces to support social judgments (Dziura & Thompson, 2020). For example, emotional value is coded by nonvisual sensory systems as early as the sensory receptors (Kryklywy, Ehlers, Anderson, & Todd, 2020). This also shows up in neural processing patterns. Recent research found that the superior colliculus possesses the necessary infrastructure to analyze the visual features that define facial expressions without additional processing stages in the visual cortex or “limbic” areas (Méndez, Celeghin, Diano, Orsenigo, Ocak, & Tamietto, 2022). For this reason, attentional engagement may not differ between spontaneous and regulated expression in holistic processing. 

This study

The complete composite face task and eye-movement technology were integrated to extend the understanding of holistic processing between regulated and spontaneous expressions. In the complete composite face task, participants were asked to judge whether the top halves of a pair of chimeric faces (see Figure 1) were the same or different in terms of expressions (Omigbodun & Cottrell, 2013). Holistic processing was inferred from an interaction of congruency and alignment: There was a stronger congruency effect when parts were aligned. In other words, the performance (measured in sensitivity, d′) was better on congruent than incongruent trials, whereas the magnitude of the congruency effect was reduced when parts were misaligned (Richler, Cheung, & Gauthier, 2011a; Richler, Cheung, & Gauthier, 2011b; Richler, Mack, Palmeri, & Gauthier, 2011c). The eye movements of the participants are also recorded during the process. Based on previous research, we hypothesized that (1) the authenticity of expressions affects holistic processing, and regulated expression (vs. spontaneous expression) may have a different extent of holistic processing; and (2) attentional engagement may not differ between spontaneous and regulated expression in holistic processing. 

Figure 1.

 

Examples of experimental stimuli from the spontaneous (up) and regulated (down) condition. Target was the upper part of the face. From left to right: congruent (top-same/bottom-same, top-different/bottom-different) and incongruent (top-different/bottom-same, top-same/bottom-different); the first is the reference stimulus.

Methods

Participants

The sample size was estimated as n = 48 using G*power 3.1 for power calculations with 95% confidence intervals (CIs), expectations of a 0.25 effect size, α set to 0.05, and a 2 × 2 × 2 repeated-measures analysis of variance (ANOVA). Accordingly, 62 participants took part in the study (12 men; M_age = 20.11; SD = 2.39; age range, 17–26 years), following the study of Curby, Entenman, and Fleming (2016). All participants were right handed and had normal or corrected-to-normal vision. All participants provided informed consent and were paid $3 for their participation. All procedures performed in the studies were in accordance with the ethical standards of the institutional and/or national research committee and in accordance with the tenets of the Declaration of Helsinki and its later amendments or comparable ethical standards. 

Apparatus

Eye movements were recorded with an EyeLink 1000 Plus desk-mounted eye tracker (SR Research, Ottawa, ON, Canada) with a 0.1 spatial resolution and 1000-Hz sample rate. Stimuli were presented on a 19-inch cathode-ray tube monitor with a 60-Hz refresh rate and a resolution of 1024 × 768. The distance between the participant's eyes and the screen was approximately 60 cm. A chin rest was used to minimize head movements. Participants’ right eyes were monitored. The experiment was programmed using Experiment Builder 1.10.1630 (SR Research), and eye tracking data were preprocessed using Data Viewer 3.1.1 (SR Research). 

Design

The experimental design was 2 × 2 × 2 (expression type × alignment × congruency) in this study. The independent variables included expression type, alignment, and congruence. The dependent variables included accuracy, d′, reaction times, and eye movement indexes (total fixation duration, fixation counts, and first fixation duration of the entire face and the top of the face), intended for judging whether the top half of the two expression pictures were the same. 

Materials

Making two types of expressions

Given the need for ecological validity, standardized stimulus materials were made based on previous studies (Buck et al., 2017). In total, 80 college students (40 men, 40 women; age range, 20–23 years) were randomly recruited to make facial expressions. Each individual signed a consent form before participating in the task. The participants were asked to sit 55 cm from a computer screen. Twelve pictures were presented through PES (Psychology Experiment System) integrated lab using E-Prime software (Psychology Software Tools, Pittsburgh, PA). A Carl Zeiss Tessar HD 1080p computer-integrated camera (ZEISS, Oberkochen, Germany) was used to capture facial expression videos. Simultaneously, the experimenter captured the participants’ facial expressions in real time through a separate computer. 

The spontaneous and regulated expressions of this study correspond to the spontaneous and regulated expressions in Buck et al. (2017). In the spontaneous expression condition, participants were filmed viewing emotionally loaded images and yielding spontaneous expressions. In the regulated expressions condition, participants were asked to pose an opposite expression—for example, making an unpleasant expression while seeing a pleasant image. Finally, 32 regulated expressions (16 men, 16 women) and 32 spontaneous expressions (16 men, 16 women) were selected. Each type of expression included 16 positive and 16 negative emotional pictures. In addition, 32 participants rated valence and authenticity on a scale from 1 (negative valence or more spontaneous) to 7 (positive valence or more regulated). There was a significant difference in valence, M_positive = 5.56, M_negative = 3.72, t(62) = 10.58, p < 0.001, Cohen's d = 2.64, and in authenticity, M_spontaneous = 4.23, M_regulated = 5.00, t(62) = −5.16, p < 0.001, Cohen's d = 1.30. Photoshop CS6 software (Adobe, San Jose, CA) was used to divide the face pictures into top and bottom halves (Richler et al. 2011c). 

Expressions used in complete composite-face task

The selected expressions were generated with a blank space of five pixels between one-half of a positive expression picture and the other half of a negative expression picture. A total of 8 types (aligned-congruent-same, aligned-congruent-different, misaligned-incongruent-same, misaligned-incongruent-different) with same identities and 64 pictures were formed from the face combinations of spontaneous condition (examples were shown in Figure 1). The face pictures in the regulated condition were combined in the same way. In the misaligned face pictures, to offset the position effect, the top half of the face was shifted by halves of the width of the image to the left or to the right. The size of all stimuli was 290 × 513 (the width and height) pixels in this study. 

Procedure

The complete composite-face task was used to conduct the experiment (see Figure 2). Participants were asked to judge whether the top halves of a pair of chimeric faces were the same or different in terms of facial expressions. In order to examine the function of facial expressions, the identities of the two faces were the same. 

Figure 2.

 

An example of spontaneous expression in the congruent-aligned condition.

Participants completed the main session after finishing a few practice trials and understanding the task. This began with a nine-point calibration and validation of the eye tracker. The formal experiments commenced after successful calibration. In the procedure, a fixation point (“+”) was first presented on the screen. The participants were asked to fixate at the + sign. The study face picture was followed by a blank screen for 500 ms, and then the target face picture was further presented. Participants were asked to press a button to judge whether the top halves of the two pictures were the same as soon as possible. If the top halves of the two pictures were the same, the participants pressed “F” on the keyboard; otherwise, they pressed “J.” Participants could respond as soon as the second image appeared; they had 3000 ms to respond. Further, the 500-ms blank screen was presented. 

The experiment included four randomized blocks. Each block included eight randomized trials in the practice experiment and 128 randomized trials in the formal experiment. Furthermore, the participants took a 5-minute break between the blocks. A calibration preceded each block. During the test, the head of each participant was fixed; no restriction was placed on the range of eye movement. 

Results

Data from four participants with unsuccessful calibrations, three participants with abnormal fixation, and seven participants with poor performance (mean accuracy less than 65%) (Chen & Cheung, 2021) were excluded, leaving 48 effective participants. The results were entered into a 2 × 2 × 2 repeated-measures ANOVA. The cleaned data for this study can be accessed at https://osf.io/3ar79/?view_only=f905f105b4144f52be76f24e3b9b8a37. 

Calculation of d′

According to signal detection theory (Green, Swets, & Emmerich, 1966; Omigbodun & Cottrell, 2013), in which the top of the correct report was the same as the hit, the top of the error report was the same as the false alarms, and d′ = Z_hit – Z_{false alarms}; d′ was calculated under each condition. ANOVA revealed a significant main effect of alignment, F(1, 47) = 18.22, p < 0.001, \( \eta^{2}_p \) = 0.28, 95% CI = 0.22 to 0.62, and congruency, F(1, 47) = 256.77, p < 0.001, \( \eta^{2}_p \) = 0.85, 95% CI = 1.56 to 2.00, but there was no significant main effect of expression type, F(1, 47) = 0.04, p = 0.835. The interaction effect of alignment and congruency was significant, F(1, 47) = 35.36, p < 0.001, \( \eta^{2}_p \) = 0.43. In the aligned condition, d′ was higher when the stimuli were congruent (M = 3.80, SE = 0.19) compared to the incongruent condition, M = 1.51, SE = 0.09, F(1, 47) = 204.60, p < 0.001, \( \eta^{2}_p \) = 0.81, 95% CI = 1.97 to 2.61, and d′ was also higher in the misaligned condition when the stimuli were congruent (M = 3.71, SE = 0.15) compared to the incongruent condition, M = 2.44, SE = 0.09, F(1, 47) = 116.90, p < 0.001, \( \eta^{2}_p \) = 0.71, 95% CI = 1.03 to 1.51. 

Furthermore, the interaction of expression type × alignment × congruency was significant (see Figure 3), F(1, 47) = 20.64, p < 0.001, \( \eta^{2}_p \) = 0.31. This suggests that the magnitude of holistic processing was influenced by the type of facial expressions. This interaction was further explored in separate ANOVAs for each expression. For spontaneous expressions, the interaction between alignment and congruency was not significant, F(1, 47) = 0.70, p = 0.408, and showed only the main effect of alignment, F(1, 47) = 6.21, p = 0.016, \( \eta^{2}_p \) = 0.12, and congruency, F(1, 47) = 257.36, p < 0.001, \( \eta^{2}_p \) = 0.85. For regulated expressions, the interaction between alignment and congruency was significant, F(1, 47) = 55.04, p < 0.001, \( \eta^{2}_p \) = 0.54; the main effects of alignment, F(1, 47) = 15.63, p < 0.001, \( \eta^{2}_p \) = 0.25, and congruency, F(1, 47) = 100.82, p < 0.001, \( \eta^{2}_p \) = 0.68, were also significant. 

Figure 3.

 

The d' in detecting spontaneous and regulated expressions under different alignment and congruency conditions. Error bars are standard errors. *p < 0.05; ***p < 0.001.

Reaction times

ANOVA showed a significant main effect of alignment, F(1, 47) = 57.26, p < 0.001, \( \eta^{2}_p \) = 0.55, 95% CI = 48.66 to 83.90, and congruency, F(1, 47) = 177.21, p < 0.001, \( \eta^{2}_p \) = 0.79, 95% CI = 44.03 to 59.71, but no significant main effect of expression type, F(1, 47) = 1.34, p = 0.253 (Table 1). Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 73.72, p < 0.001, \( \eta^{2}_p \) = 0.61. Reaction times (RTs) in the aligned condition were longer than in the misaligned condition in both the congruent and incongruent conditions. RTs in the congruent condition were shorter than in the incongruent condition in both the aligned and misaligned conditions. This indicated that there was holistic processing revealed in RTs. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.24, p = 0.630. 

Eye movements

Eye movements of test faces were examined. Based on Wang (2018), rectangular zones were adopted to divide the region of interest, and the whole face was regarded as the region of interest used for analysis. A total of 200 trials (accounting for 0.95% of the total data) were excluded from the absence of all eye movement data (any eye movement data at all times and mean saccade amplitude). Descriptive statistical results are shown in Table 2. 

Table 1.

 

Means and standard errors of the RTs for expression, alignment, and congruency.

 

Table 2.

 

Means and standard errors of the eye movements for expression, alignment, and congruency of the whole face.

 

Total fixation duration

ANOVA showed a significant main effect of alignment, F(1, 47) = 53.51, p < 0.001, \( \eta^{2}_p \) = 0.53, 95% CI = 39.21 to 68.95, and congruency, F(1, 47) = 192.32, p < 0.001, \( \eta^{2}_p \) = 0.80, 95% CI = 39.90 to 53.43, but no significant main effect of expression type, F(1, 47) = 0.79, p = 0.379. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 89.15, p < 0.001, \( \eta^{2}_p \) = 0.66. In the aligned condition, the total fixation duration (TFD) in the congruent condition (M = 688.23, SE = 14.56) was shorter than that in the incongruent condition, M = 766.13, SE = 16.66, F(1, 47) = 201.96, p < 0.001, \( \eta^{2}_p \) = 0.81, 95% CI = 66.87 to 88.92. Similarly, in the misaligned condition, the TFD in the congruent condition (M = 665.39, SE = 13.01) was also shorter than that in the incongruent condition, M = 680.82, SE = 13.26, F(1, 47) = 16.44, p < 0.001, \( \eta^{2}_p \) = 0.26, 95% CI = 7.77 to 23.09. This indicated that there was holistic processing revealed in TFD. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.29, p = 0.595. 

Fixation counts

ANOVA showed a significant main effect of alignment, F(1, 47) = 50.08, p < 0.001, \( \eta^{2}_p \) = 0.52, 95% CI = 0.17 to 0.30, and congruency, F(1, 47) = 102.43, p < 0.001, \( \eta^{2}_p \) = 0.69, 95% CI = 0.12 to 0.18, but no significant main effect of expression type, F(1, 47) = 0.27, p = 0.607. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 39.94, p < 0.001, \( \eta^{2}_p \) = 0.44. In the aligned condition, the fixation counts (FCs) in the congruent condition (M = 2.44, SE = 0.08) were fewer than those in the incongruent condition, M = 2.68, SE = 0.10, F(1, 47) = 90.34, p < 0.001, \( \eta^{2}_p \) = 0.66, 95% CI = 0.18 to 0.28. Similarly, in the misaligned condition, the FCs in the congruent condition (M = 2.29, SE = 0.07) were fewer than those in the incongruent condition, M = 2.36, SE = 0.08, F(1, 47) = 19.80, p < 0.001, \( \eta^{2}_p \) = 0.30, 95% CI = 0.04 to 0.10. This indicates that holistic processing was revealed in the FCs. The interaction effect of expression type × alignment × congruency was insignificant, F(1, 47) = 0.60, p = 0.442. 

First fixation duration

ANOVA showed a significant main effect of alignment, F(1, 47) = 4.86, p = 0.032, \( \eta^{2}_p \) = 0.09, 95% CI = 0.87 to 18.85, but no significant main effect of congruency, F(1, 47) = 0.29, p = 0.590, and expression type, F(1, 47) = 0.22, p = 0.641. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 12.82, p = 0.001, \( \eta^{2}_p \) = 0.21. In the aligned condition, the first fixation durations (FFDs) in the congruent condition (M = 345.35, SE = 13.71) were shorter than those in the incongruent condition, M = 352.88, SE = 14.65, F(1, 47) = 4.80, p = 0.033, \( \eta^{2}_p \) = 0.09, 95% CI = 0.62 to 14.45. In the misaligned condition, the FFDs in the congruent condition (M = 361.44, SE = 0.13.55) were longer than those in the incongruent condition, M = 356.50, SE = 13.91, F(1, 47) = 4.34, p = 0.043, \( \eta^{2}_p \) = 0.09, 95% CI = 0.17 to 9.72. This indicates that holistic processing was revealed in the FFDs. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.14, p = 0.711. Then, we analyzed the eye movement on the top of the face to investigate the difference between spontaneous and regulated expressions. Descriptive statistical results are shown in Table 3. 

Table 3.

 

Means and standard errors for expression, alignment, and congruency of top area.

 

Total fixation duration

ANOVA showed that there was no significant main effect of expression type, F(1, 47) = 1.43, p = 0.237, but the main effects of alignment, F(1, 47) = 17.78, p < 0.001, \( \eta^{2}_p \) = 0.28, and congruency, F(1, 47) = 69.92, p < 0.001, \( \eta^{2}_p \) = 0.60, were significant. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 58.98, p < 0.001, \( \eta^{2}_p \) = 0.56. In the aligned condition, the total fixation durations (TFDs) in the congruent condition (M = 671.08, SE = 19.26) were shorter than those in the incongruent condition, M = 752.61, SE = 22.08, 95% CI = 67.40 to 95.67. In the misaligned condition, the TFDs in the congruent condition (M = 655.57, SE = 13.37) were not significantly different from those of the incongruent condition, M = 663.22, SE = 13.94, 95% CI = −7.10 to 22.41. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.08, p = 0.783. 

Fixation counts

ANOVA showed that there was no significant main effect of Expression Type, F(1, 47) = 1.71, p = 0.197; however, the main effects of alignment, F(1, 47) = 22.95, p < 0.001, \( \eta^{2}_p \) = 0.33, and congruency, F(1, 47) = 75.60, p < 0.001, \( \eta^{2}_p \) = 0.62, were significant. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 43.38, p < 0.001, \( \eta^{2}_p \) = 0.48. In the aligned condition, the fixation counts (FCs) in the congruent condition (M = 2.37, SE = 0.09) were shorter than those in the incongruent condition, M = 2.62, SE = 0.10, 95% CI = 0.20 to 0.30. In the misaligned condition, the FCs in the congruent condition (M = 2.24, SE = 0.07) were shorter than those in the incongruent condition, M = 2.29, SE = 0.08, 95% CI = 0.01 to 0.09. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.01, p = 0.913. 

First fixation duration

ANOVA showed that there was no significant main effect of expression type, F(1, 47) = 0.17, p = 0.682, or congruency, F(1, 47) = 0.60, p = 0.442; however, the main effect of alignment was significant, F(1, 47) = 6.55, p = 0.014, \( \eta^{2}_p \) = 0.12. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 12.45, p = 0.001, \( \eta^{2}_p \) = 0.21. In the aligned condition, the first fixation durations (FFDs) in the congruent condition (M = 334.78, SE = 14.35) were shorter than those in the incongruent condition, M = 343.22, SE = 15.30, 95% CI = 1.26 to 15.62. In the misaligned condition, the FFDs in the congruent condition (M = 355.37, SE = 13.43) were greater than those in the incongruent condition, M = 350.55, SE = 13.57, 95% CI = 0.24 to 9.42. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.45, p = 0.508. 

Discussion

By integrating the complete composite-face task and eye movement technology, this study investigated the differences in holistic processing between regulated and spontaneous expressions. The results show that both regulated and spontaneous facial expressions were processed holistically. Regarding regulated expression, there was a congruency effect with the aligned condition. Moreover, the congruency effect diminished when parts were misaligned. Unlike the regulated expression, the congruency effect of the spontaneous expression did not diminish in the misaligned condition. Moreover, the regulated and spontaneous expressions did not differ in reaction times and attentional engagement. These findings support our hypothesis. Below, we discuss our findings in the order of the hypotheses. 

Holistic processing of regulated and spontaneous expressions

To the best of our knowledge, this study is the first to explore the holistic processing of regulated and spontaneous expressions. The congruency effect confirmed that both regulated and spontaneous facial expressions are processed holistically. Specifically, the regulated expression showed that the congruency effect of the aligned condition was more significant than that of the misaligned condition, indicating automatic holistic processing. However, regarding the spontaneous expression, the congruency effect was observed equally for both the aligned and misaligned conditions. This suggests that the spontaneous expression showed no typical holistic processing (Gauthier, Klaiman, & Schultz, 2009). Accordingly, individuals may lack sensitivity to configuring spontaneous expressions. Some researchers hold that evolutionary pressure makes humans more sensitive to others’ fake cues (Evans & Cruse, 2004; Yan et al., 2016). It is believed that individuals who can camouflage social signals earn more survival and reproduction opportunities and have evolutionary advantages (Yan et al., 2016). Because the perception of facial expressions is not the perception of the original emotional state but the emotional value assigned to them, making them prominent and perceiving priority (Kryklywy et al., 2020; Pourtois, Schettino, & Vuilleumier, 2013). On the other hand, facial expression processing is more likely to be holistic when the parts of the face conflict, according to Tanaka et al. (2012). Perhaps spontaneous expressions are more coordinated than regulated expressions. Thus, the spontaneous expressions did not reflect typical holistic processing. 

Attentional engagement of the regulated and spontaneous expressions in holistic processing

According to eye movement results, the difference in holistic processing between the regulated and spontaneous expressions was insignificant. This indicates that, compared with spontaneous expressions, the processing of regulated expressions requires no extra attention. It further provides evidence for a better understanding of the recognition mechanism of judging the regulated and spontaneous expressions in holistic processing. Furthermore, we analyzed eye movement differences around the upper face and found no difference in holistic processing; therefore, the perceived differences between regulated and spontaneous expressions may not result from attention. In holistic processing, all object parts were encoded obligatorily (Richler, Palmeri, & Gauthier, 2012). It allowed individuals to process and recognize expressions quickly and efficiently. Thus, the characteristic of attentional engagement of regulated and spontaneous expressions could help better detect changes in expression for adaptive and evolutionary purposes. Similarly, studies have noted that emotions can be identified within 17 ms of exposure time, and holistic processing can be achieved (Campbell & Tanaka, 2020; Gregory et al., 2021; Liu & Tanaka, 2019; Tanaka & Xu, 2018). Such findings suggest that the extent of holistic processing between regulated and spontaneous expressions is similar in attentional engagement. 

The fact that processing regulated expressions requires no extra attentional engagement compared with spontaneous expressions may be attributable to the emotion schemas in the visual cortex. Patterns of human visual cortex activity encode emotion category–related model outputs and can decode multiple categories of emotional experience so that rich, category-specific visual features can be reliably mapped to distinct emotions. Furthermore, they are coded in distributed representations within the human visual system. The activation of emotion schemas in the visual cortex provides a quick and possibly automatic way of triggering downstream emotional responses without deliberative or top–down conceptual processes (Kragel, Reddan, LaBar, & Wager, 2019). Consistent with previous studies on judging regulated and spontaneous expressions, participants of the present study did not spend more time on the eyes or mouth in the regulated expressions compared with the spontaneous expressions (Perron & Roy-Charland, 2013). Thus, because the authenticity of expression schemas is embedded in the human visual system, this method of activation and representation of emotion objects may result in the attentional engagement of judging regulated and spontaneous expressions being undifferentiated. In other words, individuals’ visual systems may be automatically distinguished between regulated and spontaneous expressions. 

Limitations and future directions

First, this study examined only positive and negative expressions; other complex expressions were not used in the composite-face task. Future research should examine more complex expressions to understand better how individuals recognize regulated and spontaneous expressions. Understanding how contextual factors influence performance on composite paradigms will inform investigations into the perceptual origins of holistic processing (Murphy et al., 2016; Richler et al., 2011a). Second, the present study uses static faces to examine the holistic processing of expressions, whereas the faces we typically encounter outside the lab are dynamic. A recent study has shown that moving faces were processed holistically (Favelle, Tobin, Piepers, Burke, & Robbins, 2015). Future research should examine the relationship between dynamic expressions and holistic face processing. Third, our study does not clarify the neural mechanism of expression processing. Based on the current findings, further investigations on the neural mechanisms in holistic processing may be noteworthy. 

Conclusion

Both regulated and spontaneous expressions were processed holistically; however, holistic processing strategies between the two types of expressions were different. Regulated expressions exhibited a more automatic holistic processing strategy, whereas spontaneous expressions showed comparable interference from irrelevant parts regardless of the alignment. Furthermore, no significant difference between the two types of expressions in reaction times and attentional characteristics in holistic processing was observed. These results suggest that holistic processing strategies of regulated and spontaneous expressions differ; however, they do not differ in attentional engagement. Therefore, individuals’ visual systems may automatically distinguish expressions to achieve effective social goals. 

Acknowledgments

Supported by a grant from the Natural Science Foundation of Shandong Province, China (ZR2020MC220). 

Commercial relationships: none. 

Corresponding author: Juncai Sun. 

Email: [email protected]. 

Address: School of Psychology, Qufu Normal University, no. 57 Jingxuan West Road, Qufu, Shandong 273165, China.