Open Access
Article  |   March 2023
Holistic processing and visual characteristics of regulated and spontaneous expressions
Author Affiliations
  • Juncai Sun
    School of Psychology, Qufu Normal University, Qufu, China
    sunjuncai266@163.com
  • Tiantian Dong
    Department of Psychology, Shanghai Normal University, Shanghai, China
    18865474552@163.com
  • Ping Liu
    Department of Psychology, Shaoxing University, Shaoxing, China
    429819721@qq.com
Journal of Vision March 2023, Vol.23, 6. doi:https://doi.org/10.1167/jov.23.3.6
  • Views
  • PDF
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Juncai Sun, Tiantian Dong, Ping Liu; Holistic processing and visual characteristics of regulated and spontaneous expressions. Journal of Vision 2023;23(3):6. https://doi.org/10.1167/jov.23.3.6.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

The rapid and efficient recognition of facial expressions is crucial for adaptive behaviors, and holistic processing is one of the critical processing methods to achieve this adaptation. Therefore, this study integrated the effects and attentional characteristics of the authenticity of facial expressions on holistic processing. The results show that both regulated and spontaneous expressions were processed holistically. However, the spontaneous expression details did not indicate typical holistic processing, with the congruency effect observed equally for aligned and misaligned conditions. No significant difference between the two expressions was observed in terms of reaction times and eye movement characteristics (i.e., total fixation duration, fixation counts, and first fixation duration). These findings suggest that holistic processing strategies differ between the two expressions. Nevertheless, the difference was not reflected in attentional engagement.

Introduction
Facial expressions display rich social signals that guide interpersonal communication, such as emotion and intention (Savage & Lipp, 2015). Based on the behavioral ecology view of facial displays, facial expressions as social tools serve as lead signs to contingent action in social interaction (Crivelli & Fridlund, 2018). That is, facial expressions are a means of achieving social influence, not merely representations of internal states (Parkinson, 2021). Therefore, facial expression perception is crucial for social interaction. 
However, people not only express genuine emotions but also regulate their facial expressions (Crivelli, Carrera, & Fernández-Dols, 2015; Yan, Wang, Zhang, Song, & Sun, 2016). In the interaction process, instead of reliable emotional states, facial expressions tend to display specific intentions deliberately (Jia, Wang, Hu, Webster, & Li, 2020). Therefore, understanding the authenticity of facial expressions has become an important yet challenging task in human behavioral research (Bartlett, Hager, Ekman, & Sejnowski, 1999; Jia et al., 2020). To the best of our knowledge, the research on processing the authenticity of expressions is limited. Hence, this study aimed to investigate the holistic processing of different authenticity expressions and their attentional characteristics. Fundamentally, it not only helps us understand the nature of the authenticity of expressions but also contributes to fresh insights into holistic processing. 
Expression processing strategies
Studies on facial expression indicate that individuals generally adopt a holistic processing strategy to evaluate facial expressions (Guo, 2012; Omigbodun & Cottrell, 2013). Although few researchers would disagree that the recognition of facial identity is holistic, there is less consensus on how the perception of different facial expressions is based on the holistic perception of the entire face (Tanaka, Kaiser, Butler, & Le Grand, 2012). 
Many researchers have observed a holistic processing of expression (Flack et al., 2015; Omigbodun & Cottrell, 2013). For example, Tanaka et al. (2012) found that facial expression processing is holistic when there is a clash between parts of a facial expression (e.g., angry–happy composite) but is analytic or parts based when there is little or no conflict between the parts (e.g., normal happy face). Holistic processing appears at a very early stage, as fast as 17 ms (Liu & Tanaka, 2019; Tanaka & Xu, 2018; Tanaka et al., 2012). Recently, researchers discovered evidence of viewpoint-invariant gaze patterns in facial expression appraisal, demonstrating that holistic mechanisms manifest during early perceptual stages (Gregory, Tanaka, & Liu, 2021; Guo & Shaw, 2015). Accordingly, the conflicting expressions may be more likely to be processed as holistic, and holistic processing of expressions perhaps occurs at an early stage. 
Processing of regulated and spontaneous expressions
During social interaction, not all facial expressions reflect actual emotions. Individuals can consciously regulate and suppress their emotions and express unfelt emotions according to their intention (Crivelli et al., 2015; Yan et al., 2016). For example, people typically laugh not only when they are genuinely happy but also when they hide sadness to maintain harmonious interpersonal relationships (Ambadar, Cohn, & Reed, 2009; Niedenthal, Mermillod, Maringer, & Hess, 2010). Thus, such behavior raises concerns about expression authenticity. 
Genuine expressions are coupled with emotional states. Be that as it may, regulated expressions are not coupled with the corresponding emotion and reflect the deception intent while ignoring the emotional state (Ekman & Rosenberg, 2005). Therefore, discerning the authenticity of expression is vital to ensure that subsequent behavior is appropriate (Zloteanu, Krumhuber, & Richardson, 2018). Misidentifying expressions could result in adverse outcomes. For example, approaching an individual who is smiling but is angry can lead to an avoidable confrontation (McLellan, Johnston, Dalrymple-Alford, & Porter, 2010). Indeed, some evidence has suggested some differences between regulated and spontaneous expressions in social perception. For example, it was observed that individuals prefer to trust and work with others who exhibit a spontaneous expression (Centorrino, Djemai, Hopfensitz, Milinski, & Seabright, 2015; Johnston, Miles, & Macrae, 2009). Therefore, authenticity may significantly contribute to face processing as an essential dimension of expressions. 
To shed light on this question, this study examined two types of expressions from the perspective of expressive authenticity—namely, regulated expression, which conveys the opposite of the original emotion, and spontaneous expression, which is a natural expression of emotion (Buck, Powers, & Hull, 2017). Previous studies have demonstrated that facial cues (e.g., race, sex, expression intensity) influence expression processing (Craig, Zhang, & Lipp, 2017; Murphy, Gray, & Cook, 2016). However, it is currently unclear how the expressions are processed to extract the authenticity information. Indubitably, it is necessary to understand the processing of regulated and spontaneous expressions to better understand the nature of expression processing. Hence, the present study used a holistic processing paradigm to explore holistic processing. 
Holistic processing is the ability or tendency to process overall facial features, which is usually evaluated using composite-face paradigms (Omigbodun & Cottrell, 2013; Richler & Gauthier, 2014). The composite-face task is either a partial design or a complete design (Gauthier & Bukach, 2007). Compared to the partial design, the complete design can eliminate the interference of response bias and has a better effect size (Richler & Gauthier, 2013; Richler & Gauthier, 2014). The complete design has been examined and validated extensively; therefore, this study used the complete composite-face task. In the complete design, the top and bottom face halves of a chimeric face could be either identical or different. If the top and bottom face halves are identical or entirely different, it is a congruent condition; otherwise, it is incongruent. 
Eye movements
Eye movements are functional in face processing (Altair, Tanaka, & Dawel, 2021; Henderson, Williams, & Falk, 2005; Schwarzer, Huber, & Dümmler, 2005; Tanaka, Liu, & Martin, 2021) and are often used to study the holistic processing of faces (Bombari, Mast, & Lobmaier, 2009; de Heering, Rossion, Turati, & Simion, 2008; van Belle, Lefèvre, Laguesse, Busigny, De Graef, Verfaillie, & Rossion, 2010). Specifically, the total fixation duration, fixation counts, and first fixation duration in eye movements are effective indicators of attentional engagement (Calvo, Gutiérrez-García, Avero, & Lundqvist, 2013; de Heering et al., 2008). Thus, this study used these three indicators to explore the attentional engagement differences between regulated and spontaneous expressions in holistic processing. 
Moreover, researchers have noted that humans can rapidly encode information from faces to support social judgments (Dziura & Thompson, 2020). For example, emotional value is coded by nonvisual sensory systems as early as the sensory receptors (Kryklywy, Ehlers, Anderson, & Todd, 2020). This also shows up in neural processing patterns. Recent research found that the superior colliculus possesses the necessary infrastructure to analyze the visual features that define facial expressions without additional processing stages in the visual cortex or “limbic” areas (Méndez, Celeghin, Diano, Orsenigo, Ocak, & Tamietto, 2022). For this reason, attentional engagement may not differ between spontaneous and regulated expression in holistic processing. 
This study
The complete composite face task and eye-movement technology were integrated to extend the understanding of holistic processing between regulated and spontaneous expressions. In the complete composite face task, participants were asked to judge whether the top halves of a pair of chimeric faces (see Figure 1) were the same or different in terms of expressions (Omigbodun & Cottrell, 2013). Holistic processing was inferred from an interaction of congruency and alignment: There was a stronger congruency effect when parts were aligned. In other words, the performance (measured in sensitivity, d′) was better on congruent than incongruent trials, whereas the magnitude of the congruency effect was reduced when parts were misaligned (Richler, Cheung, & Gauthier, 2011a; Richler, Cheung, & Gauthier, 2011b; Richler, Mack, Palmeri, & Gauthier, 2011c). The eye movements of the participants are also recorded during the process. Based on previous research, we hypothesized that (1) the authenticity of expressions affects holistic processing, and regulated expression (vs. spontaneous expression) may have a different extent of holistic processing; and (2) attentional engagement may not differ between spontaneous and regulated expression in holistic processing. 
Figure 1.
 
Examples of experimental stimuli from the spontaneous (up) and regulated (down) condition. Target was the upper part of the face. From left to right: congruent (top-same/bottom-same, top-different/bottom-different) and incongruent (top-different/bottom-same, top-same/bottom-different); the first is the reference stimulus.
Figure 1.
 
Examples of experimental stimuli from the spontaneous (up) and regulated (down) condition. Target was the upper part of the face. From left to right: congruent (top-same/bottom-same, top-different/bottom-different) and incongruent (top-different/bottom-same, top-same/bottom-different); the first is the reference stimulus.
Methods
Participants
The sample size was estimated as n = 48 using G*power 3.1 for power calculations with 95% confidence intervals (CIs), expectations of a 0.25 effect size, α set to 0.05, and a 2 × 2 × 2 repeated-measures analysis of variance (ANOVA). Accordingly, 62 participants took part in the study (12 men; Mage = 20.11; SD = 2.39; age range, 17–26 years), following the study of Curby, Entenman, and Fleming (2016). All participants were right handed and had normal or corrected-to-normal vision. All participants provided informed consent and were paid $3 for their participation. All procedures performed in the studies were in accordance with the ethical standards of the institutional and/or national research committee and in accordance with the tenets of the Declaration of Helsinki and its later amendments or comparable ethical standards. 
Apparatus
Eye movements were recorded with an EyeLink 1000 Plus desk-mounted eye tracker (SR Research, Ottawa, ON, Canada) with a 0.1 spatial resolution and 1000-Hz sample rate. Stimuli were presented on a 19-inch cathode-ray tube monitor with a 60-Hz refresh rate and a resolution of 1024 × 768. The distance between the participant's eyes and the screen was approximately 60 cm. A chin rest was used to minimize head movements. Participants’ right eyes were monitored. The experiment was programmed using Experiment Builder 1.10.1630 (SR Research), and eye tracking data were preprocessed using Data Viewer 3.1.1 (SR Research). 
Design
The experimental design was 2 × 2 × 2 (expression type × alignment × congruency) in this study. The independent variables included expression type, alignment, and congruence. The dependent variables included accuracy, d′, reaction times, and eye movement indexes (total fixation duration, fixation counts, and first fixation duration of the entire face and the top of the face), intended for judging whether the top half of the two expression pictures were the same. 
Materials
Making two types of expressions
Given the need for ecological validity, standardized stimulus materials were made based on previous studies (Buck et al., 2017). In total, 80 college students (40 men, 40 women; age range, 20–23 years) were randomly recruited to make facial expressions. Each individual signed a consent form before participating in the task. The participants were asked to sit 55 cm from a computer screen. Twelve pictures were presented through PES (Psychology Experiment System) integrated lab using E-Prime software (Psychology Software Tools, Pittsburgh, PA). A Carl Zeiss Tessar HD 1080p computer-integrated camera (ZEISS, Oberkochen, Germany) was used to capture facial expression videos. Simultaneously, the experimenter captured the participants’ facial expressions in real time through a separate computer. 
The spontaneous and regulated expressions of this study correspond to the spontaneous and regulated expressions in Buck et al. (2017). In the spontaneous expression condition, participants were filmed viewing emotionally loaded images and yielding spontaneous expressions. In the regulated expressions condition, participants were asked to pose an opposite expression—for example, making an unpleasant expression while seeing a pleasant image. Finally, 32 regulated expressions (16 men, 16 women) and 32 spontaneous expressions (16 men, 16 women) were selected. Each type of expression included 16 positive and 16 negative emotional pictures. In addition, 32 participants rated valence and authenticity on a scale from 1 (negative valence or more spontaneous) to 7 (positive valence or more regulated). There was a significant difference in valence, Mpositive = 5.56, Mnegative = 3.72, t(62) = 10.58, p < 0.001, Cohen's d = 2.64, and in authenticity, Mspontaneous = 4.23, Mregulated = 5.00, t(62) = −5.16, p < 0.001, Cohen's d = 1.30. Photoshop CS6 software (Adobe, San Jose, CA) was used to divide the face pictures into top and bottom halves (Richler et al. 2011c). 
Expressions used in complete composite-face task
The selected expressions were generated with a blank space of five pixels between one-half of a positive expression picture and the other half of a negative expression picture. A total of 8 types (aligned-congruent-same, aligned-congruent-different, misaligned-incongruent-same, misaligned-incongruent-different) with same identities and 64 pictures were formed from the face combinations of spontaneous condition (examples were shown in Figure 1). The face pictures in the regulated condition were combined in the same way. In the misaligned face pictures, to offset the position effect, the top half of the face was shifted by halves of the width of the image to the left or to the right. The size of all stimuli was 290 × 513 (the width and height) pixels in this study. 
Procedure
The complete composite-face task was used to conduct the experiment (see Figure 2). Participants were asked to judge whether the top halves of a pair of chimeric faces were the same or different in terms of facial expressions. In order to examine the function of facial expressions, the identities of the two faces were the same. 
Figure 2.
 
An example of spontaneous expression in the congruent-aligned condition.
Figure 2.
 
An example of spontaneous expression in the congruent-aligned condition.
Participants completed the main session after finishing a few practice trials and understanding the task. This began with a nine-point calibration and validation of the eye tracker. The formal experiments commenced after successful calibration. In the procedure, a fixation point (“+”) was first presented on the screen. The participants were asked to fixate at the + sign. The study face picture was followed by a blank screen for 500 ms, and then the target face picture was further presented. Participants were asked to press a button to judge whether the top halves of the two pictures were the same as soon as possible. If the top halves of the two pictures were the same, the participants pressed “F” on the keyboard; otherwise, they pressed “J.” Participants could respond as soon as the second image appeared; they had 3000 ms to respond. Further, the 500-ms blank screen was presented. 
The experiment included four randomized blocks. Each block included eight randomized trials in the practice experiment and 128 randomized trials in the formal experiment. Furthermore, the participants took a 5-minute break between the blocks. A calibration preceded each block. During the test, the head of each participant was fixed; no restriction was placed on the range of eye movement. 
Results
Data from four participants with unsuccessful calibrations, three participants with abnormal fixation, and seven participants with poor performance (mean accuracy less than 65%) (Chen & Cheung, 2021) were excluded, leaving 48 effective participants. The results were entered into a 2 × 2 × 2 repeated-measures ANOVA. The cleaned data for this study can be accessed at https://osf.io/3ar79/?view_only=f905f105b4144f52be76f24e3b9b8a37
Calculation of d
According to signal detection theory (Green, Swets, & Emmerich, 1966; Omigbodun & Cottrell, 2013), in which the top of the correct report was the same as the hit, the top of the error report was the same as the false alarms, and d′ = ZhitZfalse alarms; d′ was calculated under each condition. ANOVA revealed a significant main effect of alignment, F(1, 47) = 18.22, p < 0.001, \( \eta^{2}_p \) = 0.28, 95% CI = 0.22 to 0.62, and congruency, F(1, 47) = 256.77, p < 0.001, \( \eta^{2}_p \) = 0.85, 95% CI = 1.56 to 2.00, but there was no significant main effect of expression type, F(1, 47) = 0.04, p = 0.835. The interaction effect of alignment and congruency was significant, F(1, 47) = 35.36, p < 0.001, \( \eta^{2}_p \) = 0.43. In the aligned condition, d′ was higher when the stimuli were congruent (M = 3.80, SE = 0.19) compared to the incongruent condition, M = 1.51, SE = 0.09, F(1, 47) = 204.60, p < 0.001, \( \eta^{2}_p \) = 0.81, 95% CI = 1.97 to 2.61, and d′ was also higher in the misaligned condition when the stimuli were congruent (M = 3.71, SE = 0.15) compared to the incongruent condition, M = 2.44, SE = 0.09, F(1, 47) = 116.90, p < 0.001, \( \eta^{2}_p \) = 0.71, 95% CI = 1.03 to 1.51. 
Furthermore, the interaction of expression type × alignment × congruency was significant (see Figure 3), F(1, 47) = 20.64, p < 0.001, \( \eta^{2}_p \) = 0.31. This suggests that the magnitude of holistic processing was influenced by the type of facial expressions. This interaction was further explored in separate ANOVAs for each expression. For spontaneous expressions, the interaction between alignment and congruency was not significant, F(1, 47) = 0.70, p = 0.408, and showed only the main effect of alignment, F(1, 47) = 6.21, p = 0.016, \( \eta^{2}_p \) = 0.12, and congruency, F(1, 47) = 257.36, p < 0.001, \( \eta^{2}_p \) = 0.85. For regulated expressions, the interaction between alignment and congruency was significant, F(1, 47) = 55.04, p < 0.001, \( \eta^{2}_p \) = 0.54; the main effects of alignment, F(1, 47) = 15.63, p < 0.001, \( \eta^{2}_p \) = 0.25, and congruency, F(1, 47) = 100.82, p < 0.001, \( \eta^{2}_p \) = 0.68, were also significant. 
Figure 3.
 
The d' in detecting spontaneous and regulated expressions under different alignment and congruency conditions. Error bars are standard errors. *p < 0.05; ***p < 0.001.
Figure 3.
 
The d' in detecting spontaneous and regulated expressions under different alignment and congruency conditions. Error bars are standard errors. *p < 0.05; ***p < 0.001.
Reaction times
ANOVA showed a significant main effect of alignment, F(1, 47) = 57.26, p < 0.001, \( \eta^{2}_p \) = 0.55, 95% CI = 48.66 to 83.90, and congruency, F(1, 47) = 177.21, p < 0.001, \( \eta^{2}_p \) = 0.79, 95% CI = 44.03 to 59.71, but no significant main effect of expression type, F(1, 47) = 1.34, p = 0.253 (Table 1). Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 73.72, p < 0.001, \( \eta^{2}_p \) = 0.61. Reaction times (RTs) in the aligned condition were longer than in the misaligned condition in both the congruent and incongruent conditions. RTs in the congruent condition were shorter than in the incongruent condition in both the aligned and misaligned conditions. This indicated that there was holistic processing revealed in RTs. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.24, p = 0.630. 
Eye movements
Eye movements of test faces were examined. Based on Wang (2018), rectangular zones were adopted to divide the region of interest, and the whole face was regarded as the region of interest used for analysis. A total of 200 trials (accounting for 0.95% of the total data) were excluded from the absence of all eye movement data (any eye movement data at all times and mean saccade amplitude). Descriptive statistical results are shown in Table 2
Table 1.
 
Means and standard errors of the RTs for expression, alignment, and congruency.
Table 1.
 
Means and standard errors of the RTs for expression, alignment, and congruency.
Table 2.
 
Means and standard errors of the eye movements for expression, alignment, and congruency of the whole face.
Table 2.
 
Means and standard errors of the eye movements for expression, alignment, and congruency of the whole face.
Total fixation duration
ANOVA showed a significant main effect of alignment, F(1, 47) = 53.51, p < 0.001, \( \eta^{2}_p \) = 0.53, 95% CI = 39.21 to 68.95, and congruency, F(1, 47) = 192.32, p < 0.001, \( \eta^{2}_p \) = 0.80, 95% CI = 39.90 to 53.43, but no significant main effect of expression type, F(1, 47) = 0.79, p = 0.379. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 89.15, p < 0.001, \( \eta^{2}_p \) = 0.66. In the aligned condition, the total fixation duration (TFD) in the congruent condition (M = 688.23, SE = 14.56) was shorter than that in the incongruent condition, M = 766.13, SE = 16.66, F(1, 47) = 201.96, p < 0.001, \( \eta^{2}_p \) = 0.81, 95% CI = 66.87 to 88.92. Similarly, in the misaligned condition, the TFD in the congruent condition (M = 665.39, SE = 13.01) was also shorter than that in the incongruent condition, M = 680.82, SE = 13.26, F(1, 47) = 16.44, p < 0.001, \( \eta^{2}_p \) = 0.26, 95% CI = 7.77 to 23.09. This indicated that there was holistic processing revealed in TFD. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.29, p = 0.595. 
Fixation counts
ANOVA showed a significant main effect of alignment, F(1, 47) = 50.08, p < 0.001, \( \eta^{2}_p \) = 0.52, 95% CI = 0.17 to 0.30, and congruency, F(1, 47) = 102.43, p < 0.001, \( \eta^{2}_p \) = 0.69, 95% CI = 0.12 to 0.18, but no significant main effect of expression type, F(1, 47) = 0.27, p = 0.607. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 39.94, p < 0.001, \( \eta^{2}_p \) = 0.44. In the aligned condition, the fixation counts (FCs) in the congruent condition (M = 2.44, SE = 0.08) were fewer than those in the incongruent condition, M = 2.68, SE = 0.10, F(1, 47) = 90.34, p < 0.001, \( \eta^{2}_p \) = 0.66, 95% CI = 0.18 to 0.28. Similarly, in the misaligned condition, the FCs in the congruent condition (M = 2.29, SE = 0.07) were fewer than those in the incongruent condition, M = 2.36, SE = 0.08, F(1, 47) = 19.80, p < 0.001, \( \eta^{2}_p \) = 0.30, 95% CI = 0.04 to 0.10. This indicates that holistic processing was revealed in the FCs. The interaction effect of expression type × alignment × congruency was insignificant, F(1, 47) = 0.60, p = 0.442. 
First fixation duration
ANOVA showed a significant main effect of alignment, F(1, 47) = 4.86, p = 0.032, \( \eta^{2}_p \) = 0.09, 95% CI = 0.87 to 18.85, but no significant main effect of congruency, F(1, 47) = 0.29, p = 0.590, and expression type, F(1, 47) = 0.22, p = 0.641. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 12.82, p = 0.001, \( \eta^{2}_p \) = 0.21. In the aligned condition, the first fixation durations (FFDs) in the congruent condition (M = 345.35, SE = 13.71) were shorter than those in the incongruent condition, M = 352.88, SE = 14.65, F(1, 47) = 4.80, p = 0.033, \( \eta^{2}_p \) = 0.09, 95% CI = 0.62 to 14.45. In the misaligned condition, the FFDs in the congruent condition (M = 361.44, SE = 0.13.55) were longer than those in the incongruent condition, M = 356.50, SE = 13.91, F(1, 47) = 4.34, p = 0.043, \( \eta^{2}_p \) = 0.09, 95% CI = 0.17 to 9.72. This indicates that holistic processing was revealed in the FFDs. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.14, p = 0.711. Then, we analyzed the eye movement on the top of the face to investigate the difference between spontaneous and regulated expressions. Descriptive statistical results are shown in Table 3
Table 3.
 
Means and standard errors for expression, alignment, and congruency of top area.
Table 3.
 
Means and standard errors for expression, alignment, and congruency of top area.
Total fixation duration
ANOVA showed that there was no significant main effect of expression type, F(1, 47) = 1.43, p = 0.237, but the main effects of alignment, F(1, 47) = 17.78, p < 0.001, \( \eta^{2}_p \) = 0.28, and congruency, F(1, 47) = 69.92, p < 0.001, \( \eta^{2}_p \) = 0.60, were significant. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 58.98, p < 0.001, \( \eta^{2}_p \) = 0.56. In the aligned condition, the total fixation durations (TFDs) in the congruent condition (M = 671.08, SE = 19.26) were shorter than those in the incongruent condition, M = 752.61, SE = 22.08, 95% CI = 67.40 to 95.67. In the misaligned condition, the TFDs in the congruent condition (M = 655.57, SE = 13.37) were not significantly different from those of the incongruent condition, M = 663.22, SE = 13.94, 95% CI = −7.10 to 22.41. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.08, p = 0.783. 
Fixation counts
ANOVA showed that there was no significant main effect of Expression Type, F(1, 47) = 1.71, p = 0.197; however, the main effects of alignment, F(1, 47) = 22.95, p < 0.001, \( \eta^{2}_p \) = 0.33, and congruency, F(1, 47) = 75.60, p < 0.001, \( \eta^{2}_p \) = 0.62, were significant. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 43.38, p < 0.001, \( \eta^{2}_p \) = 0.48. In the aligned condition, the fixation counts (FCs) in the congruent condition (M = 2.37, SE = 0.09) were shorter than those in the incongruent condition, M = 2.62, SE = 0.10, 95% CI = 0.20 to 0.30. In the misaligned condition, the FCs in the congruent condition (M = 2.24, SE = 0.07) were shorter than those in the incongruent condition, M = 2.29, SE = 0.08, 95% CI = 0.01 to 0.09. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.01, p = 0.913. 
First fixation duration
ANOVA showed that there was no significant main effect of expression type, F(1, 47) = 0.17, p = 0.682, or congruency, F(1, 47) = 0.60, p = 0.442; however, the main effect of alignment was significant, F(1, 47) = 6.55, p = 0.014, \( \eta^{2}_p \) = 0.12. Furthermore, the interaction effect of alignment and congruency was significant, F(1, 47) = 12.45, p = 0.001, \( \eta^{2}_p \) = 0.21. In the aligned condition, the first fixation durations (FFDs) in the congruent condition (M = 334.78, SE = 14.35) were shorter than those in the incongruent condition, M = 343.22, SE = 15.30, 95% CI = 1.26 to 15.62. In the misaligned condition, the FFDs in the congruent condition (M = 355.37, SE = 13.43) were greater than those in the incongruent condition, M = 350.55, SE = 13.57, 95% CI = 0.24 to 9.42. The interaction effect of expression type × alignment × congruency was not significant, F(1, 47) = 0.45, p = 0.508. 
Discussion
By integrating the complete composite-face task and eye movement technology, this study investigated the differences in holistic processing between regulated and spontaneous expressions. The results show that both regulated and spontaneous facial expressions were processed holistically. Regarding regulated expression, there was a congruency effect with the aligned condition. Moreover, the congruency effect diminished when parts were misaligned. Unlike the regulated expression, the congruency effect of the spontaneous expression did not diminish in the misaligned condition. Moreover, the regulated and spontaneous expressions did not differ in reaction times and attentional engagement. These findings support our hypothesis. Below, we discuss our findings in the order of the hypotheses. 
Holistic processing of regulated and spontaneous expressions
To the best of our knowledge, this study is the first to explore the holistic processing of regulated and spontaneous expressions. The congruency effect confirmed that both regulated and spontaneous facial expressions are processed holistically. Specifically, the regulated expression showed that the congruency effect of the aligned condition was more significant than that of the misaligned condition, indicating automatic holistic processing. However, regarding the spontaneous expression, the congruency effect was observed equally for both the aligned and misaligned conditions. This suggests that the spontaneous expression showed no typical holistic processing (Gauthier, Klaiman, & Schultz, 2009). Accordingly, individuals may lack sensitivity to configuring spontaneous expressions. Some researchers hold that evolutionary pressure makes humans more sensitive to others’ fake cues (Evans & Cruse, 2004; Yan et al., 2016). It is believed that individuals who can camouflage social signals earn more survival and reproduction opportunities and have evolutionary advantages (Yan et al., 2016). Because the perception of facial expressions is not the perception of the original emotional state but the emotional value assigned to them, making them prominent and perceiving priority (Kryklywy et al., 2020; Pourtois, Schettino, & Vuilleumier, 2013). On the other hand, facial expression processing is more likely to be holistic when the parts of the face conflict, according to Tanaka et al. (2012). Perhaps spontaneous expressions are more coordinated than regulated expressions. Thus, the spontaneous expressions did not reflect typical holistic processing. 
Attentional engagement of the regulated and spontaneous expressions in holistic processing
According to eye movement results, the difference in holistic processing between the regulated and spontaneous expressions was insignificant. This indicates that, compared with spontaneous expressions, the processing of regulated expressions requires no extra attention. It further provides evidence for a better understanding of the recognition mechanism of judging the regulated and spontaneous expressions in holistic processing. Furthermore, we analyzed eye movement differences around the upper face and found no difference in holistic processing; therefore, the perceived differences between regulated and spontaneous expressions may not result from attention. In holistic processing, all object parts were encoded obligatorily (Richler, Palmeri, & Gauthier, 2012). It allowed individuals to process and recognize expressions quickly and efficiently. Thus, the characteristic of attentional engagement of regulated and spontaneous expressions could help better detect changes in expression for adaptive and evolutionary purposes. Similarly, studies have noted that emotions can be identified within 17 ms of exposure time, and holistic processing can be achieved (Campbell & Tanaka, 2020; Gregory et al., 2021; Liu & Tanaka, 2019; Tanaka & Xu, 2018). Such findings suggest that the extent of holistic processing between regulated and spontaneous expressions is similar in attentional engagement. 
The fact that processing regulated expressions requires no extra attentional engagement compared with spontaneous expressions may be attributable to the emotion schemas in the visual cortex. Patterns of human visual cortex activity encode emotion category–related model outputs and can decode multiple categories of emotional experience so that rich, category-specific visual features can be reliably mapped to distinct emotions. Furthermore, they are coded in distributed representations within the human visual system. The activation of emotion schemas in the visual cortex provides a quick and possibly automatic way of triggering downstream emotional responses without deliberative or top–down conceptual processes (Kragel, Reddan, LaBar, & Wager, 2019). Consistent with previous studies on judging regulated and spontaneous expressions, participants of the present study did not spend more time on the eyes or mouth in the regulated expressions compared with the spontaneous expressions (Perron & Roy-Charland, 2013). Thus, because the authenticity of expression schemas is embedded in the human visual system, this method of activation and representation of emotion objects may result in the attentional engagement of judging regulated and spontaneous expressions being undifferentiated. In other words, individuals’ visual systems may be automatically distinguished between regulated and spontaneous expressions. 
Limitations and future directions
First, this study examined only positive and negative expressions; other complex expressions were not used in the composite-face task. Future research should examine more complex expressions to understand better how individuals recognize regulated and spontaneous expressions. Understanding how contextual factors influence performance on composite paradigms will inform investigations into the perceptual origins of holistic processing (Murphy et al., 2016; Richler et al., 2011a). Second, the present study uses static faces to examine the holistic processing of expressions, whereas the faces we typically encounter outside the lab are dynamic. A recent study has shown that moving faces were processed holistically (Favelle, Tobin, Piepers, Burke, & Robbins, 2015). Future research should examine the relationship between dynamic expressions and holistic face processing. Third, our study does not clarify the neural mechanism of expression processing. Based on the current findings, further investigations on the neural mechanisms in holistic processing may be noteworthy. 
Conclusion
Both regulated and spontaneous expressions were processed holistically; however, holistic processing strategies between the two types of expressions were different. Regulated expressions exhibited a more automatic holistic processing strategy, whereas spontaneous expressions showed comparable interference from irrelevant parts regardless of the alignment. Furthermore, no significant difference between the two types of expressions in reaction times and attentional characteristics in holistic processing was observed. These results suggest that holistic processing strategies of regulated and spontaneous expressions differ; however, they do not differ in attentional engagement. Therefore, individuals’ visual systems may automatically distinguish expressions to achieve effective social goals. 
Acknowledgments
Supported by a grant from the Natural Science Foundation of Shandong Province, China (ZR2020MC220). 
Commercial relationships: none. 
Corresponding author: Juncai Sun. 
Email: sunjuncai266@163.com. 
Address: School of Psychology, Qufu Normal University, no. 57 Jingxuan West Road, Qufu, Shandong 273165, China. 
References
Altair, T., Tanaka, J., & Dawel, A. (2021). Emotion recognition of perceived genuine or posed expressions using web-based eye tracking. Journal of Vision, 21(9), 2983, https://doi.org/10.1167/jov.21.9.2983. [CrossRef]
Ambadar, Z., Cohn, J. F., & Reed, L. I. (2009). All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed. Journal of Nonverbal Behavior, 33(1), 17–34. [CrossRef] [PubMed]
Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36(2), 253–263. [CrossRef] [PubMed]
Bombari, D., Mast, F. W., & Lobmaier, J. S. (2009). Featural, configural, and holistic face-processing strategies evoke different scan patterns. Perception, 38(10), 1508–1521. [CrossRef] [PubMed]
Buck, R., Powers, S. R., & Hull, K. S. (2017). Measuring emotional and cognitive empathy using dynamic, naturalistic, and spontaneous emotion displays. Emotion, 17(7), 1120–1136. [CrossRef] [PubMed]
Calvo, M. G., Gutiérrez-García, A., Avero, P., & Lundqvist, D. (2013). Attentional mechanisms in judging genuine and fake smiles: Eye-movement patterns. Emotion, 13(4), 792–802. [CrossRef] [PubMed]
Campbell, A., & Tanaka, J. W. (2020). Face identity processing at 33 ms and 100 ms with 4 ms of stimulus exposure. Journal of Vision, 20(11), 1478–1478, https://doi.org/10.1167/jov.20.11.1478. [CrossRef]
Centorrino, S., Djemai, E., Hopfensitz, A., Milinski, M., & Seabright, P. (2015). Honest signaling in trust interactions: Smiles rated as genuine induce trust and signal higher earning opportunities. Evolution and Human Behavior, 36(1), 8–16. [CrossRef]
Chen, W., & Cheung, O. S. (2021). Flexible face processing: Holistic processing of facial identity is modulated by task-irrelevant facial expression. Vision Research, 178, 18–27. [CrossRef] [PubMed]
Craig, B. M., Zhang, J., & Lipp, O. V. (2017). Facial race and sex cues have a comparable influence on emotion recognition in Chinese and Australian participants. Attention, Perception, and Psychophysics, 79(7), 2212–2223. [CrossRef]
Crivelli, C., Carrera, P., & Fernández-Dols, J.-M. (2015). Are smiles a sign of happiness? Spontaneous expressions of judo winners. Evolution and Human Behavior, 36(1), 52–58. [CrossRef]
Crivelli, C., & Fridlund, A. J. (2018). Facial displays are tools for social influence. Trends in Cognitive Sciences, 22(5), 388–399. [CrossRef] [PubMed]
Curby, K. M., Entenman, R. J., & Fleming, J. T. (2016). Holistic face perception is modulated by experience-dependent perceptual grouping. Attention, Perception and Psychophysics, 78(5), 1392–1404. [CrossRef]
de Heering, A., Rossion, B., Turati, C., & Simion, F. (2008). Holistic face processing can be independent of gaze behavior: Evidence from the composite face illusion. Journal of Neuropsychology, 2(1), 183–195. [CrossRef] [PubMed]
Dziura, S. L., & Thompson, J. C. (2020). Temporal dynamics of the neural representation of social relationships. Journal of Neuroscience, 40(47), 9078–9087. [CrossRef]
Ekman, P., & Rosenberg, E. L. (2005). What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford: Oxford University Press.
Evans, D., & Cruse, P. (2004). Emotion, evolution, and rationality. Oxford: Oxford University Press.
Favelle, S., Tobin, A., Piepers, D., Burke, D., & Robbins, R. A. (2015). Dynamic composite faces are processed holistically. Vision Research, 112, 26–32. [CrossRef] [PubMed]
Flack, T. R., Andrews, T. J., Hymers, M., Al-Mosaiwi, M., Marsden, S. P., Strachan, J. W., & Young, A. W. (2015). Responses in the right posterior superior temporal sulcus show a feature-based response to facial expression. Cortex, 69, 14–23. [CrossRef] [PubMed]
Gauthier, I., & Bukach, C. (2007). Should we reject the expertise hypothesis? Cognition, 103(2), 322–333. [CrossRef] [PubMed]
Gauthier, I., Klaiman, C., & Schultz, R. T. (2009). Face composite effects reveal abnormal face processing in autism spectrum disorders. Vision Research, 49(4), 470–478. [CrossRef] [PubMed]
Green, D. M., Swets, J. A., & Emmerich, D. S. (1966). Signal detection theory and psychophysics. Malabar, FL: Robert E. Krieger.
Gregory, E., Tanaka, J. W., & Liu, X. (2021). Emotional gist: The rapid perception of facial expressions. Cognition and Emotion, 35(2), 385–392. [CrossRef]
Guo, K. (2012). Holistic gaze strategy to categorize facial expression of varying intensities. PLoS One, 7(8), e42585. [CrossRef] [PubMed]
Guo, K., & Shaw, H. (2015). Face in profile view reduces perceived facial expression intensity: An eye-tracking study. Acta Psychologica, 155, 19–28. [CrossRef] [PubMed]
Henderson, J. M., Williams, C. C., & Falk, R. J. (2005). Eye movements are functional during face learning. Memory and Cognition, 33(1), 98–106. [CrossRef]
Jia, S., Wang, S., Hu, C., Webster, P., & Li, X. (2020). Detection of genuine and posed facial expressions of emotion: A review. Frontiers in Psychology, 11, 580287. [CrossRef] [PubMed]
Johnston, L., Miles, L., & Macrae, C. N. (2009). Why are you smiling at me? Social functions of enjoyment and non-enjoyment smiles. British Journal of Social Psychology, 49(1), 107–127. [CrossRef]
Kragel, P. A., Reddan, M., LaBar, K. S., & Wager, T. D. (2019). Emotion schemas are embedded in the human visual system. Science Advances, 5, eaaw4358. [CrossRef] [PubMed]
Kryklywy, J. H., Ehlers, M. R., Anderson, A. K., & Todd, R. M. (2020). From architecture to evolution: Multisensory evidence of decentralized emotion. Trends in Cognitive Sciences, 24(11), 916–929. [CrossRef] [PubMed]
Liu, X., & Tanaka, J. W. (2019). Holistic perception of faces in 17 milliseconds: Evidence from three measures. Journal of Vision, 19(10), 92, https://doi.org/10.1167/19.10.92. [CrossRef]
McLellan, T., Johnston, L., Dalrymple-Alford, J., & Porter, R. (2010). Sensitivity to genuine versus posed emotion specified in facial displays. Cognition and Emotion, 24(8), 1277–1292. [CrossRef]
Méndez, C. A., Celeghin, A., Diano, M., Orsenigo, D., Ocak, B., & Tamietto, M. (2022). A deep neural network model of the primate superior colliculus for emotion recognition. Philosophical Transactions of the Royal Society B: Biological Sciences, 377(1863), 20210512. [CrossRef]
Murphy, J., Gray, K. L. H., & Cook, R. (2016). The composite face illusion. Psychonomic Bulletin and Review, 24(2), 245–261. [CrossRef]
Niedenthal, P. M., Mermillod, M., Maringer, M., & Hess, U. (2010). The Simulation of Smiles (SIMS) model: Embodied simulation and the meaning of facial expression. Behavioral and Brain Sciences, 33(6), 447–448.
Omigbodun, A., & Cottrell, G. (2013). Is facial expression processing holistic? Proceedings of the Annual Meeting of the Cognitive Science Society, 35(35), 3187–3192.
Parkinson, B. (2021). Heart to heart: A relation-alignment approach to emotion's social effects. Emotion Review, 13(2), 78–89. [CrossRef]
Perron, M., & Roy-Charland, A. (2013). Analysis of eye movements in the judgment of enjoyment and non-enjoyment smiles. Frontiers in Psychology, 4, 659. [CrossRef] [PubMed]
Pourtois, G., Schettino, A., & Vuilleumier, P. (2013). Brain mechanisms for emotional influences on perception and attention: What is magic and what is not. Biological Psychology, 92(3), 492–512. [CrossRef] [PubMed]
Richler, J. J., Cheung, O. S., & Gauthier, I. (2011a). Beliefs alter holistic face processing if response bias is not taken into account. Journal of Vision, 11(13):17, 74–76, https://doi.org/10.1167/11.13.17. [CrossRef]
Richler, J. J., Cheung, O. S., & Gauthier, I. (2011b). Holistic processing predicts face recognition. Psychological Science, 22(4), 464–471. [CrossRef] [PubMed]
Richler, J. J., & Gauthier, I. (2013). When intuition fails to align with data: A reply to Rossion (2013). Visual Cognition, 21(2), 254–276. [CrossRef]
Richler, J. J., & Gauthier, I. (2014). A meta-analysis and review of holistic face processing. Psychological Bulletin, 140(5), 1281–1302. [CrossRef] [PubMed]
Richler, J. J., Mack, M. L., Palmeri, T. J., & Gauthier, I. (2011c). Inverted faces are (eventually) processed holistically. Vision Research, 51(3), 333–342. [CrossRef] [PubMed]
Richler, J. J., Palmeri, T. J., & Gauthier, I. (2012). Meanings, mechanisms, and measures of holistic processing. Frontiers in Psychology, 3(53), 553. [PubMed]
Savage, R. A., & Lipp, O. V. (2015). The effect of face inversion on the detection of emotional faces in visual search. Cognition and Emotion, 29(6), 972–991. [CrossRef]
Schwarzer, G., Huber, S., & Dümmler, T. (2005). Gaze behavior in analytical and holistic face processing. Memory and Cognition, 33(2), 344–354. [CrossRef]
Tanaka, J. W., Kaiser, M. D., Butler, S., & Le Grand, R. (2012). Mixed emotions: Holistic and analytic perception of facial expressions. Cognition and Emotion, 26(6), 961–977. [CrossRef]
Tanaka, J., Liu, X., & Martin, J. (2021). Investigating the time course of face perception using web-based eye tracking. Journal of Vision, 21(9), 2965, https://doi.org/10.1167/jov.21.9.2965. [CrossRef]
Tanaka, J., & Xu, B. (2018). Holistic gist: The speed of holistic face processing. Journal of Vision, 18(10), 166–166, https://doi.org/10.1167/18.10.166. [CrossRef]
Van Belle, G., Lefèvre, P., Laguesse, R., Busigny, T., De Graef, P., Verfaillie, K., et al. (2010). Feature-based processing of personally familiar faces in prosopagnosia: Evidence from eye-gaze contingency. Behavioural Neurology, 23(4), 255–257. [CrossRef] [PubMed]
Wang, S. (2018). Face size biases emotion judgment through eye movement. Scientific Reports, 8(1), 317. [CrossRef] [PubMed]
Yan, L. L., Wang, Z., Zhang, Z. J., Song, S. W., & Sun, Y. H. (2016). Face holistic processing is modulated by perceptual experience. Acta Psychologica Sinica, 48(10), 1199–1209. [CrossRef]
Zloteanu, M., Krumhuber, E. G., & Richardson, D. C. (2018). Detecting genuine and deliberate displays of surprise in static and dynamic faces. Frontiers in Psychology, 9, 1184. [CrossRef] [PubMed]
Figure 1.
 
Examples of experimental stimuli from the spontaneous (up) and regulated (down) condition. Target was the upper part of the face. From left to right: congruent (top-same/bottom-same, top-different/bottom-different) and incongruent (top-different/bottom-same, top-same/bottom-different); the first is the reference stimulus.
Figure 1.
 
Examples of experimental stimuli from the spontaneous (up) and regulated (down) condition. Target was the upper part of the face. From left to right: congruent (top-same/bottom-same, top-different/bottom-different) and incongruent (top-different/bottom-same, top-same/bottom-different); the first is the reference stimulus.
Figure 2.
 
An example of spontaneous expression in the congruent-aligned condition.
Figure 2.
 
An example of spontaneous expression in the congruent-aligned condition.
Figure 3.
 
The d' in detecting spontaneous and regulated expressions under different alignment and congruency conditions. Error bars are standard errors. *p < 0.05; ***p < 0.001.
Figure 3.
 
The d' in detecting spontaneous and regulated expressions under different alignment and congruency conditions. Error bars are standard errors. *p < 0.05; ***p < 0.001.
Table 1.
 
Means and standard errors of the RTs for expression, alignment, and congruency.
Table 1.
 
Means and standard errors of the RTs for expression, alignment, and congruency.
Table 2.
 
Means and standard errors of the eye movements for expression, alignment, and congruency of the whole face.
Table 2.
 
Means and standard errors of the eye movements for expression, alignment, and congruency of the whole face.
Table 3.
 
Means and standard errors for expression, alignment, and congruency of top area.
Table 3.
 
Means and standard errors for expression, alignment, and congruency of top area.
×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×