How positive emotional content overrules perceptual history effects: Hysteresis in emotion recognition

Abstract

The human visual system is constantly processing multiple and often conflicting sensory cues to make perceptual decisions. Given the nonlinear nature of emotion recognition, this often leads to different percepts of the same physical facial expression. Moreover, the state of the emotion recognition system might depend on the trajectory of temporal context, potentially leading to a phenomenon known as perceptual hysteresis. Here, we aimed to explore temporal context-related mechanisms underlying perceptual hysteresis during emotion recognition. We hypothesized that dependence on recent perceptual experience might reveal important clues about the role of short-term memory on the perception of emotional stimuli. Behavioral data were acquired using reality-based, changing emotion expressions morphed from a source to a target emotion with different valences, always passing through a neutral expression. Participants identified the onset and offset of what they perceived as the neutral expression interval. Our results showed that current perception of emotional expression is affected by recent temporal context, thus revealing perceptual hysteresis. We also found a relation between recent perceptual history effects and stimulus emotional Content: The positive valence of the stimulus emotional content appeared to abolish perceptual history effects, whereas negatively loaded stimuli induced clear short-term memory effects and positive hysteresis. Our findings show direct competition between recent perceptual experience and stimulus emotional content during decision making, which affects the formation of current percepts in emotion recognition.

Introduction

People frequently shift from one emotional state to another in order to accurately adapt themselves to the environment in which they are integrated (Young, Rowland, Calder, Etcoff, Seth, & Perrett, 1997; Filipowicz, Barsade, & Melwani, 2011). These dynamic transitions, essential for effective and adaptive social interactions (Mayer & Salovey, 1990), often result in a combination of expressions of both emotional states giving rise to different interpretations of the same facial expression. 

Emotion recognition has properties that may be useful to be studied using dynamic systems approaches (Barrett, Ochsner, & Gross, 2007; Witherington & Crichton, 2007), in particular when emotions emerge in real-time contexts and undergo change across time. Dynamic systems are time dependent, and their current state is determined not only by the external inputs but also by their recent state (Luenberger, 1979). As such, they might have more than one possible path in the evolution of the system and more than one possible outcome. In terms of visual perception, the outcomes are often associated with nonlinear phase transitions, characteristic of ambiguous states having more than one perceptual interpretation (Attneave, 1971). 

The output of a dynamic system may also depend on the variation trajectory of an input control parameter; this particular history-based dependence is known as hysteresis (Warburg, 1881). Extensively described in magnetism (Warburg, 1881; Stoner & Wohlfarth, 1948; Jiles & Atherton, 1986), this time-dependence concept has raised interest in the field of cognitive neuroscience, particularly with regard to visual perception (Williams, Phillips, & Sekuler, 1986; Kobayashi & Hara, 1993; Kleinschmidt, Büchel, Hutton, Friston, & Frackowiak, 2002). Hysteresis has accordingly been described as reflecting the current state dependence on temporal context (Dubé, 1997). The phenomenon occurs when the transition point between two perceptual states is dependent on the direction of the visual input change and appears to play a role in the disambiguation of multistable ambiguous stimuli (Kleinschmidt et al., 2002; Hock & Schöner, 2010; Sacharin, Sander, & Scherer, 2012; Witthoft, Sha, Winawer, & Kiani, 2018). Depending on temporal context, perceptual history mechanisms might contribute to the persistence on the original percept (Kleinschmidt et al., 2002; Pearson & Brascamp, 2008; Hsu & Wu, 2019). This instantiates positive hysteresis, which maintains the current percept, even when the stimulus parameters favor an alternative one. 

In the positive hysteresis mechanism, a switch to that different percept is delayed (Ziemann, Tergau, Wassermann, Wischer, Hildebrandt, & Paulus, 1998; Monterosso & Ainslie, 1999; Sanes & Donoghue, 2000; Blake, Sobel, & Gilroy, 2003; Chen & He, 2004; Fischer & Whitney, 2014; Fritsche, Mostert, & de Lange, 2017). If the switch occurs earlier than the actual physical changes of the stimulus favoring the alternative one, the negative hysteresis occurs. Hysteresis has therefore been characterized in terms of two types of behavior, positive or negative, reported to occur in several self-organizing dynamical systems and modeled following a unified perspective (Lopresti-Goodman, Turvey, & Frank, 2013; Frank, Profeta, & Harrison, 2015; Frank, 2019). When comparing the transition point between two perceptual states based on opposite trajectories, positive hysteresis (or simply hysteresis, as many authors choose to refer to it) occurs when the switch value between two states is higher for ascending than for descending variations of the input variable. It has been related to memory (persistence) mechanisms due to recent perceptual experience (Kleinschmidt et al., 2002; Hock & Schöner, 2010; Sacharin et al., 2012; Liaci, Fischer, Atmanspacher, Heinrichs, Tebartz van Else, & Kornmeijer, 2018). Negative hysteresis describes the opposite case and has been related to adaptation/habituation mechanisms that lead to earlier switches (Webster, Kaping, Mizokami, & Duhamel, 2004; Pisarchik, Jaimes-Reátegui, Magallón-García, & Castillo-Morales, 2014; Liaci et al., 2018). 

Hysteresis in emotion recognition was first described by Kobayashi and Hara (1993). The authors demonstrated that when observing dynamically changing facial expressions, perception was dependent on the direction of the change. More recently, hysteresis has been described in the interpretation of dynamic transitions between happiness and sadness (Liaci et al., 2018; Witthoft et al., 2018) and between anger and disgust (Sacharin et al., 2012). Early research on the perception of emotional facial expressions supported by the basic theory of emotion processing mainly focused on the use of static stimuli with direct comparisons between extreme emotion expression apexes (Barrett et al., 2007; Witherington & Crichton, 2007). Nevertheless, taking into account that emotion perception is largely a dynamic process, it can be best studied within a dynamic systems framework by investigating the transition between two emotional expressions and how it passes through the origin of the space of emotions, in which the expression has neutral emotional content (Woodworth & Schlosberg, 1954; Young et al., 1997). In fact, emotions emerging in everyday life interactions are dynamic; thus, more realistic stimuli based on dynamic emotional facial expression transitions have been used in emotion recognition studies (Dubé, 1997; LaBar, Crupain, Voyvodic, & McCarthy, 2003; Sato, Kochiyama, Yoshikawa, Naito, & Matsumura, 2004; Trautmann, Fehr, & Herrmann, 2009; Sacharin et al., 2012; Kamachi, Bruce, Mukaida, Gyoba, Yoshikawa, & Akamatsu, 2013). However, most of these have been limited to changes from and to neutral expressions, disregarding transitions between different emotional states (LaBar et al., 2003; Sato et al., 2004; Webster et al., 2004; Kamachi et al., 2013). Moreover, temporal context (history) effects on the current perception have in general been overlooked (LaBar et al., 2003; Webster et al., 2004). 

Here, we studied the mechanisms underlying temporal context (trajectory) effects on the perception of realistic emotional transitions. We hypothesized that the perception of transition between two emotional states depends on recent perceptual history and its direction of variation. We also hypothesized that the comparison of these perceptual trajectories with the control case of no recent perceptual influence (no history) would help us to understand how memory mechanisms contribute to the perceptual decision on emotion recognition. Investigation of perceptual hysteresis provides an estimation of the contribution of memory mechanisms to the current percept (Maier, Wilke, Logothetis, & Leopold, 2003; Sterzer & Rees, 2008; Knapen, Brascamp, Adams, & Graf, 2009; Schwiedrzik, Ruff, Lazar, Leitner, Singer, & Melloni, 2014; Kim & Frank, 2018; Liaci et al., 2018). Short-term memory of recent perceptual experience has been identified as a relevant mechanism (Pastukhov & Braun, 2008; Olkkonen & Allred, 2014), along with long-term memory bias effects on perception, which are based on everyday experience (van Rooij, Atmanspacher, & Kornmeijer, 2016; Kim & Frank, 2018; Liaci et al., 2018). 

Our results revealed the existence of temporal context effects on the perception of emotion expressions, as we identified perceptual hysteresis in all of the changing emotion expressions tested. Moreover, by comparing the pattern of perception on each direction with the no-history situation, we found that subjects persisted on their first percept, despite the direction of stimulation, which reveals a perceptual persistence mechanism. Importantly, we were able to identify a relation between recent perceptual history effects and stimulus emotional content. When subjects first perceived a negative emotion, their perception of the target emotion of opposite (positive) valence behaved as if no recent perceptual history effects stored in short-term memory existed. Our results suggest that this prioritized processing of the stimulus emotional content over the effects of positive hysteresis unveils a competition between the two processes. The overriding of recent perceptual history in the decision-making process highlights an interaction between emotional processing and mechanisms of perceptual hysteresis. 

Methods

Participants

Twenty healthy young adults participated in Experiment 1a (11 females; mean age, 27.76 ± 4.06 years), and 17 healthy young adults participated in Experiment 1b (nine females; mean age, 27.53 ± 4.02 years). This sample size is justified by the effect sizes expected from this type of experiment. All had a normal or corrected-to-normal vision and no history of neurological or psychiatric diseases. Participants provided informed consent prior to data collection following protocols approved by the Ethics Committee of the Faculty of Medicine of the University of Coimbra, in accordance with the tenets of the Declaration of Helsinki. 

Experimental setup and apparatus

The experiment consisted of six visual stimulation runs investigating emotion recognition. For replication purposes, it was conducted under two distinct stimulus conditions (Experiment 1a and Experiment 1b). Behavioral data were acquired during the perception of dynamic transitions between pairs of emotions assessing the contribution of recent perceptual experience (three runs, 24 trials per run). As a control condition (no history), without the contribution of recent perceptual experience, we used random sequences of emotion snapshots with no gradual transitions (three runs, 108 trials per run). The order of the total sum of six runs was random for each subject. Presentation 20.1 software (Neurobehavioral Systems, Inc., Albany CA) was used to design and present the stimuli for both tasks and to collect participants’ responses. In Experiment 1a, stimuli were presented in the center of a liquid-crystal display (LCD) monitor with a refresh rate of 60 Hz and a resolution of 1024 × 768 pixels, and participants were positioned at a distance of 65 to 70 cm from the display screen. In Experiment 1b, stimuli were presented in the center of an LCD screen with a refresh rate of 60 Hz and a resolution of 1920 × 1080 pixels, and participants were positioned at a distance of 156 cm from the display screen. 

Stimuli

Frontal-view images of posed expressions of four females and three males were chosen from the Extended Cohn–Kanade Dataset (Lucey, Cohn, Kanade, Saragih, Ambadar, & Matthews, 2010). The images were cropped by a uniform rectangle (with a visual angle of 10.48° horizontally and 8.37° vertically in Experiment 1a and a visual angle of 5.05° horizontally and 4.03° vertically in Experiment 1b), allowing the preservation of not only the internal features of the faces used but also the surrounding external features. The images were posteriorly corrected for luminance levels (average luminance value of 44.76 ± 1.73 cd/m²). All stimuli were presented on a gray background (23.10 cd/m²). The posed expressions used were representative of three basic emotions (happiness, sadness, and anger) and neutral expressions. The basic emotions of fear, disgust, and surprise were deliberately not chosen because of temporal constraints and due to their potential ambiguity and morphological similarities with at least one of the remaining emotions here used (Sacharin et al., 2012). 

Dynamic transitions

Dynamic transitions consisted of a source facial expression (E1), which gradually evolved to a neutral expression (N) and posteriorly to a target emotion expression (E2). Source and target emotions were always different. Transitions were obtained by morphing sequences of digital images. The sequences of the images were generated using MorphMan 4.0 software (STOIK, Moscow, Russia) and by applying approximately 80 fiducial markers, which were densely placed in face-relevant areas, such as the eyes, mouth, and corrugator muscles (Ekman & Friesen, 1978). Morphs were chosen to allow finer control over the rate and duration of the changing expressions, as previously done by others (LaBar et al., 2003; Sato et al., 2004; Webster et al., 2004; Sato, Kochiyama, Uono, & Yoshikawa, 2010; Sacharin et al., 2012). 

In total, each dynamic emotion transition was based on an array of 81 sequential images of the same actor with a morphing step of 2.50% (40 intermediate images between each emotion and neutral expression). The image sequences were posteriorly transformed into movie clips with a frame rate of 8.89 frames/second using MATLAB (MathWorks, Natick, MA), totaling a morphing duration of 9 seconds. Each frame was presented for 113.92 ms, except for the first and last morph frames, which corresponded to the source and target emotion expressions and remained on the screen for 500 ms. As such, each dynamic transition was 10 seconds long. (An example of a dynamic transition used in the angerhappiness task is available at https://bit.ly/2DmkdKc.) 

Three pairs of emotions were used to design the dynamic transitions: anger–happiness, sadness–anger, and sadness–happiness. Two stimulation directions were defined: direction 1, where the source was the less positive emotion and the target was the more positive emotion; and direction 2, where the source was the more positive emotion and the target was the less positive (Figure 1). Both basic emotions sadness and anger are located in the negative portion of the valence axis in the two-dimensional space of valence and arousal (Russell, 1980). As such, participants were asked before starting the experiment, giving them no particular judgmental feature nor requiring them to justify their choice, which of the two emotions they considered to be more positive. The majority of them (12/20 in Experiment 1a and 10/17 in Experiment 1b) chose anger as the more positive emotion against sadness. 

Figure 1.

 

Examples of dynamic transitions between pairs of emotions used during visual stimulation. Each stimulation trial consisted of a dynamic transition between a less positive emotion and a more positive emotion (direction 1) or a transition in the opposite direction (direction 2), always passing through a neutral expression. The source emotion expression gradually evolved to a neutral expression, which posteriorly evolved to a target emotion expression. The neutral expression corresponded to the zero percent of the positive emotion (used to parameterize each morphing sequence). The examples illustrate transitions between (A) sadness and anger, (B) anger and happiness, and (C) sadness and happiness. The examples presented here were randomly chosen from the dataset.

An additional control dynamic transition was used as a comparison for the changing emotional facial expression. An intermediate neutral expression was the posed neutral expression used to design the remaining changing emotion sequences; however, both the source and target images were neutral expressions (distinct neutral expressions of the correspondent actor available in the dataset used). Thus, although the morphing and presentation parameters were the same as those used in the remaining transitions, no emotion changes were occurring. 

Control presentation of random snapshots of facial expressions

Snapshots of facial expressions were used as static control trials for the dynamic transitions between different emotions. Nine relevant images from each morphing sequence used for the dynamic transitions were selected (Table 1) and randomly presented during a fixed time of 2000 ms. Because the stimuli did not follow a sequential direction, no perceptual bias due to recent previous experience (history) was expected, thus they served as a control condition for perceptual history effects for the dynamic emotion transitions. 

Table 1.

 

Static facial expression frames selected from the complete sequence used for the dynamic transitions and the correspondent percentage of negative (E1) and positive (E2) emotion and neutral expression (N).

 

Tasks

Participants were asked to report the perceived facial expression during visual stimulation based on dynamic transitions and static figures. Perceptual reports were given via button press on both tasks. Each participant performed three runs of images presented in a random order. 

Emotion recognition based on dynamic transitions

Each run based on dynamic transitions included four dynamic transition trials of each pair of emotions in both directions of stimulation (thus, 4 repetitions × 3 pairs of emotions × 2 directions = 24 trials per run). Considering the block design within each run, control dynamic transitions were presented in an interleaved fashion for use as a baseline to transitions between different expressions (no response to emotional change expected). 

Trials were presented in a pseudorandomized order, ensuring that there were no two consecutive dynamic transitions of the same actor. At the beginning and end of each run, a black fixation cross (with a visual angle of 2.35° horizontally and 2.35° vertically in Experiment 1a and a visual angle of 1.01° horizontally and 1.01° vertically in Experiment 1b) on a gray background was presented for 10 seconds, which was also shown after every eight transitions. These periods allowed the participants to rest. An example of the experimental design of a run is shown in Figure 2. 

Figure 2.

 

Experimental design, with runs based on dynamic transitions. Each run included four dynamic transition trials of each pair of emotions in both directions of stimulation interleaved with control dynamic trials (N2NN2), which were used as baseline. Fixation crosses (FCs) were presented at the beginning and end of the run and also after every eight transitions of interest. ANH, anger–neutral–happiness; HNA, happiness–neutral–anger; SNA, sadness–neutral–anger; ANS, anger–neutral–sadness; SNH: sadness–neutral–happiness; HNS, happiness–neutral–sadness; T₀, run beginning; T_e, run ending.

Participants were instructed to identify the neutral expression interval of each dynamic transition by pressing the response button whenever they identified a change to and from a neutral facial expression (P1 and P2 moments, reflecting emotion discrimination), thus allowing the identification of the onset and offset of the interval of a neutral expression. Only one response button (keyboard spacebar) was available for identifying both P1 and P2. Participants were informed of the presence of control dynamic transitions and were instructed to press the response button if they identified a change in the neutral facial expression. Each run lasted 9.24 minutes. Participants performed one training session before starting the main experiment. 

Emotion recognition based on static facial expressions

Each static control run contained static images (no defined history) taken from the dynamic transitions between each pair of emotions: static control run 1, anger and happiness; run 2, sadness and anger; and run 3, sadness and happiness. This setup was chosen to prevent participants from becoming confused by images belonging to transitions from different pairs of emotions but having similar features. 

A run was composed of 12 repetitions of the nine selected expressions (108 trials, Table 1), each one presented for 2000 ms. Every run followed a block design interleaved with a baseline. Baseline corresponded to one of the images chosen randomly from the control dynamic transitions (transitions between neutral expressions). In these transitions, although morphing was occurring, there was no variation in emotion percentage; therefore, any 100% neutral frame could be chosen randomly as a baseline for the static control runs. 

As in runs of dynamic transitions, each static control run also started and ended with a fixation cross for 5 seconds and another two distributed in the task after every 36 repetitions of the conditions of interest. The task also required emotion discrimination. An example of the experimental design of a run is shown in Figure 3. 

Figure 3.

 

Experimental design, with static control runs. Each run included 12 repetitions of the nine selected expressions (see Table 1), which were interleaved with images chosen randomly from the control dynamic transitions that were used as baseline. Fixation crosses (FCs) were presented at the beginning and end of the run, and also after every 36 transitions of interest. T₀, run beginning; T_e, run ending.

Participants were instructed to classify the emotion perceived during each static image presented as a positive emotion expression (E⁺), a neutral expression (N), or as a negative emotion expression (E^–). Each classification was locked to a specific response button in the keyboard (right arrow, down arrow, and left arrow, respectively). Participants’ responses were collected via a button press. Each stimulation run lasted 7.70 minutes. 

Data processing

We first compared, for each pair of emotions, the perceptual switch curves in both directions of variation, in order to investigate the phenomena of perceptual hysteresis. Next, the perceptual reports curves in each direction were compared with the case of no recent perceptual influence (no defined history) to further understand the contribution of memory mechanisms (presence of persistence) to the perceptual decision on emotion recognition. A description of the metrics used and analyses performed is provided below. 

Figure 4.

 

Experiment 1a: Perceptual hysteresis on emotion recognition during dynamic transitions. Group results of the average perceptual inflection point (mean of transition points at each neutral interval border) for both directions 1 (green) and 2 (blue) and the respective static control condition (discontinuous black curve). Perceptual switch differences between the two directions are statistically significant for the pairs of emotions (B) sadness–anger (p = 0.002) and (C) sadness–happiness (p = 0.025); however, no statistically significant differences were found for the pair of emotions (A) anger–happiness (p = 0.083). A, anger; H, happiness; S, sadness. Error bars correspond to within-subject SEM.

Figure 5.

 

Experiment 1b: Perceptual hysteresis on emotion recognition during dynamic transitions. Group results of the average perceptual inflection point (mean of transition points at each neutral interval border) for both directions 1 (green) and 2 (blue) and the respective static control condition (discontinuous black curve). Statistically significant differences between both directions were found in all three pairs of emotions: (A) anger–happiness (p = 0.001); (B) sadness–anger (p = 0.0003); and (C) sadness–happiness (p = 0.001). A, anger; H, happiness; S, sadness. Error bars correspond to within-subject SEM.

Figure 6.

 

Experiment 1a: Averaged report of neutral across all participants as a function of the percent of positive emotion present in each of the images’ sequence for all three pairs of emotions: (A) anger–happiness, (B) sadness–anger, and (C) sadness–happiness. Green and blue curves correspond to directions 1 and 2, respectively, and the discontinuous black curve corresponds to the reports registered in the static control trials. The report tendency, particularly when traveling in direction 1, overlaps in the “positive region” with the reports given in the classification of the static control stimuli, when no recent perceptual history existed. Vertical lines represent the points correspondent to the nine frames used in the static control trials for which the statistical analysis was performed. Asterisks above lines denote highly significant differences between each perceptual trajectory and the control curve (p < 0.05). Error bars correspond to within-subject SEM.

Figure 7.

 

Experiment 1b: Averaged report of neutral across all participants as a function of the percent of positive emotion present in each of the images’ sequence, for all three pairs of emotions: (A) anger–happiness, (B) sadness–anger; and (C) sadness–happiness. Green and blue curves correspond to directions 1 and 2, respectively, and the discontinuous black curve corresponds to the reports registered in the static control trials. As observed in Experiment 1a, even under the different conditions of stimuli visualization in Experiment 1b, the report tendency when traveling in direction 1 overlaps in the “positive region” with the reports given in the classification of the static control trials, when no recent perceptual history existed. Asterisks above lines denote highly significant differences between each perceptual trajectory and the control curve (p < 0.05). Error bars correspond to within-subject SEM.

Metric of perceptual hysteresis

Because participants were required to identify the first and last time points in which they perceived the neutral expression, in dynamic transitions this allowed us to define an interval of what they had perceived as a neutral expression. Any trials in which only one or no response was given were excluded from analysis due to the impossibility of defining the neutral interval (5.60% of the trials in Experiment 1a and 4.65% of the trials in Experiment 1b). In the control dynamic transitions, with transitions between neutral expressions serving as comparisons for the changing emotional facial expressions, the majority had no response, as would be expected from the fact that the expressions were neutral. The number of trials in which two responses were given was lower than 10% (6.98 ± 2.01% in Experiment 1a and 4.72 ± 2.53% in Experiment 1b). The average point of each border of the neutral interval was used as the transition point between each pair of emotions for each stimulation trajectory. The difference between the perceptual switch in each direction was estimated as a metric of hysteresis. 

The percentage of neutral reports given for each static image shown in the static control runs was obtained, which allowed us to establish which frames were perceived as neutral. The position of the control black line was subsequently estimated by averaging the position of the earlier and later frames that participants reported as being neutral expressions. 

Evaluation of the influence of previous perceptual trajectories on the current percept

The percentages of neutral reports for each frame of the dynamic transitions were estimated. These findings were then averaged within and across all participants, for both directions of each pair of emotions. These reports were parameterized according to the percentage of positive emotions reported for each image of the morphing array and represented in a histogram using a bin width corresponding to the morphing step (2.50%). A Gaussian curve was subsequently fitted to the data. Average adjusted R² and root-mean-square error (RMSE) were estimated, and they revealed that the curve fit the data well (Supplementary Table S1). These curves were then compared to similar ones obtained from the emotion recognition task based on pseudorandomly ordered snapshots of static expressions (emotion recognition without the influence of recent perceptual experience). 

For each randomly presented static image, with different percentages of neutral expression and positive or negative emotions, the number of participants’ reports classifying the image as neutral was calculated. Participants were instructed to provide a double button press in case they wanted to correct any impulsive response given in these trials; thus, only the latter responses were considered. In addition, trials in which no response was registered were discarded from the analysis (8% of the trials in Experiment 1a and 5% of the trials in Experiment 1b). To assess the distribution of neutral perception over time, the percentage of the neutral reports for each image shown was calculated and averaged within all participants. Data were linearly interpolated to match the total number of frames presented on dynamic transitions for visualization purposes only. The percentage of positive emotions reported for each image of the morph sequence and the averaged reports were represented in a histogram using a bin width corresponding to the morphing step (2.50%), and a Gaussian curve was fitted to the data (Supplementary Table S1). 

Statistical analysis

Data were tested for normality using a Kolmogorov–Smirnov test with an α-level of 0.05 before running further statistical analyses. In order to analyze the significance of the hysteresis metric, the perceptual switch difference between direction 1 and direction 2 was compared using a non-parametric Wilcoxon test. The influence of the valence of emotions on perceptual history was tested by comparison of the participants’ reports of each dynamic transition trajectory with the static control case, taking into account only the points corresponding to the used frames. A Wilcoxon signed-rank test analysis was used for this purpose. Additionally, a confirmatory analysis was performed taking an area-under-the-curve approach (see Supplementary Table S2). Using the fitted parameters, the percentages of neutral reports were integrated into two different intervals of percent of positive emotions, (–100, 0) and (0, 100), for both directions of stimulation and the static control case in all three pairs of emotions tested. Distributions were compared using non-parametric Wilcoxon signed-rank tests. All statistical analyses were performed with SPSS Statistics V22.0 (IBM, Armonk, NY) and MATLAB. 

Results

Perceptual hysteresis on emotion recognition

Perceptual hysteresis was analyzed based on the difference of the perceptual switch from a source to a target emotion between two opposite stimulation morphing trajectories. We found perceptual hysteresis in both Experiments 1a and 1b (replication experiment under distinct stimulus conditions) for the three pairs of dynamic transitions tested as the mean transition point depended on the trajectory. 

In Experiment 1a (Figure 4), for the pairs of emotions sadness–anger and sadness–happiness, Wilcoxon matched-pairs signed-ranked tests revealed significant differences in the perceptual switch moment between both directions (Z = –3.173, p = 0.002 and Z = –2.240, p = 0.025, respectively), with the greater difference being found for the sadness–anger pair. For the pair anger–happiness, however, Wilcoxon signed-rank tests revealed only a very marginal near significance difference in the perceptual shifting points between the two directions tested (Z = –1.736, p = 0.083). In Experiment 1b (Figure 5), statistical significance between both directions in the perceptual switch moment was found for all three pairs of emotions: anger–happiness (Z = –3.181, p = 0.001); sadness–anger (Z = –3.621, p = 0.0003); and sadness–happiness (Z = –3.243, p = 0.001). As found previously, the greatest difference was observed for the pair sadness–anger. 

In both environments of stimuli visualization and for all three pairs of emotions, the perceptual switch occurred at a higher percentage of positive emotion when traveling from the negative toward the positive emotion, as compared to the control curve (and vice versa), suggesting persistence. Accordingly, subjects needed a greater percent of positive emotion in stimulus content, as compared to the control situation, to switch between percepts when traveling from a source emotion of negative toward a target emotion of positive valence (and vice versa), which indicates positive hysteresis. 

Enhanced processing of positive valence overrules recent perceptual experience effects

The effects of recent perceptual history on the continuous perception of dynamic emotional transitions were then directly analyzed. Participants’ reports were estimated for each frame of the presented stimulus in perceptual trajectories (with perceptual history influence) and in the static control trials (without perceptual history influence). The perceptual curves over time in Experiment 1a and in Experiment 1b are represented in Figures 6 and 7, respectively. The participants’ reports of neutral expressions were plotted as a function of the percentage of positive emotion present in each of the stimulus frames. A Gaussian curve was adjusted to fit the data, for visualization purposes only. 

Overall, in both experiments, participants tended to persist longer on the original percept, as compared with the no-history situation, independently of the direction of stimulation. When traveling from the negative-valence source emotion toward a more positive one (Figures 6 and 7, left panel), the observed differences between curves with and without the influence of recent perceptual history are mainly in the negative portion of the axis. After the perception of the neutral expression, as stimuli evolved toward the emotion of positive valence, the two curves tend to overlap. In the opposite direction of stimulation (Figures 6 and 7, right panel), differences between curves are much larger when stimuli depicting positive emotions are absent (middle right panel). 

In replication Experiment 1b, under different conditions of stimuli visualization, these effects appeared to be enhanced for all three pairs of emotions, with a striking consistency for the pair sadness–happiness in both Experiments 1a and 1b. Despite the direction of stimulation (although the effect was more prominent for direction 1), the two curves with and without the influence of recent perceptual history tended to overlap when the stimuli were mainly of positive valence. 

Discussion

In the current study, we investigated the role of dynamic temporal context effects in the perception of emotion by manipulating trajectories of facial expressions. We were interested in studying hysteresis in emotion recognition and the mechanisms underlying this phenomenon. In order to do so, we applied a visual paradigm consisting of reality-based transitions from a source to a target morphed emotion with different valences, always passing through a neutral expression. 

The participants’ perceptions of the dynamic emotional transitions tested here were found to be dependent on the direction of stimulation, thereby revealing perceptual hysteresis. Earlier reports in emotion perception have shown short-term memory effects (Kobayashi & Hara, 1993; Sacharin et al., 2012; Liaci et al., 2018; Witthoft et al., 2018; Mei, Chen, & Dong, 2019). However, most of the previous studies have mainly focused on extreme comparisons between emotional expression apexes, disregarding the dimensions within the space of emotions and hysteresis (Young et al., 1997). Our study adds to the previous findings by revealing different perceptual trajectories based on dynamic transitions between basic emotions passing through a neutral expression, thereby revealing hysteresis and competition in terms of emotional content. 

We found positive hysteresis in both experiments conducted for all three pairs of emotions tested. The perceptual switch when traveling from the source emotion toward the target emotion occurred later than in the opposite direction. This positive lag signature is one of the two variants of the hysteresis phenomenon and the most broadly studied in the field of visual perception (Webster et al., 2004; Pisarchik et al., 2014; Liaci et al., 2018). Importantly, our results revealed perceptual hysteresis between emotions widely spread in the emotional space. As such, we also have shown that perceptual hysteresis in emotion recognition is not specific of emotions that share similar features and musculature activity, as previously suggested (Sacharin et al., 2012). Here, we tested emotion recognition processes with (dynamic transitions) and without (pseudo-random static control trials) the influence of recent perceptual history. This allowed for the study of short-term memory effects (Liaci et al., 2018). 

The distribution of the perceptual reports during our static control trials (no history) revealed that the participants’ perceptual transitions in all three pairs tested encompassed a window including the point when the percent of the positive emotion was zero, when the stimuli content was fully neutral. This indicates that participants perceived either the source emotion or the target emotion, and their perception did not alternate between the two competing percepts. This reveals a stable well-behaved perceptual output, a characteristic feature of dynamic systems (Luenberger, 1979). Perception is known to be influenced by recent perceptual history (Fischer & Whitney, 2014; Fritsche et al., 2017; Hsu & Wu, 2019). This type of effect has been described to reflect a short-term memory mechanistic contribution to decision-making processes (Pastukhov & Braun, 2008; Olkkonen & Allred, 2014; Liaci et al., 2018). The rationale is that short-term memory contributes to stabilizing a percept and reducing ambiguity (Fritsche et al., 2017; Liberman, Manassi, & Whitney, 2018; Mei et al., 2019), leading to positive hysteresis. 

The contribution of memory mechanisms to perceptual decisions on emotion recognition was investigated by comparing the perceptual report curves in each direction with the case of no recent perceptual influence (no defined history). Despite differences between tasks (the dynamic task required detection of an emotional expression, and the static task involved identification of a specific emotion), in both cases the type of decision involved emotion recognition. Our results were consistent across the pairs of emotions tested and revealed that, when traveling from a source to a target emotion, participants tended to persist in the original percept (compared to the no-history control), regardless of the direction of stimulation. Such persistence is a correlate of short-term memory. This perceptual persistence effect is based on recent perceptual experience and reflects the positive lag that is typical of positive hysteresis (Hock, Kelso, & Schöner, 1993), whereby participants tend to perceive the same content overtime (Kleinschmidt et al., 2002; Pearson & Brascamp, 2008). 

The positive hysteresis effect did, however, exhibit a notable asymmetry. After the perceptual switch, the perceptual reports curves showed two different patterns of behavior depending on the stimulation trajectory. In particular, when traveling from a negative toward a positive emotion, in the “positive region” the perceptual report curves tend to overlap the control ones after the perceptual switch. As such, there is no evidence for the influence of recent perceptual history on the current percept. This asymmetry was registered in both Experiments 1a and 1b. 

The tendency to overlap the no-history curve when the stimulation trajectory was toward a positive emotion suggests an abolishment of perceptual history effects. This reveals a possible interaction between the stimulus of emotional content and recent perceptual history effects stored in short-term memory. In fact, previous studies have reported that emotional content influences decision-making processes, as emotional stimuli are indeed perceived faster and more accurately (Hansen and Hansen, 1988; Öhman, Lundqvist, & Esteves, 2001), when compared to non-emotional stimuli. 

We now show evidence that perceptual history effects are mainly evident during the recognition of emotions of negative valence. As a consequence, the processing of positive emotional information appears to be prioritized and to overrule recent perceptual experience effects stored in short-term memory, establishing current perception as if no recent perceptual history existed. This might reveal a competition between fundamental mechanisms of emotion processing and short-term memory contribution in the disambiguation process, adding to previous studies that have shown that working memory is able to modulate serial-dependent effects in emotional expression perception (Liberman et al., 2018; Hsu & Wu, 2019; Mei et al., 2019). 

Nonetheless, it is important to acknowledge some limitations of our experimental design attached to the fact that it is intrinsically difficult to match dynamic transitions (succession of snapshots) and static controls. Thus, trials in dynamic runs differed in duration from trials in the static runs. In other words, static trials had greater presentation times than the corresponding short frames in the dynamic trials. By making dynamic and static tasks more comparable, future experimental designs could overcome this limitation; for example, they could consider a static task in which participants report only whether or not the frame is neutral. Also, increasing the number of points in the control condition (no history) while maintaining a bearable experiment duration could increase the statistical power and significance of our findings. 

Future studies should be conducted to reveal the role of distinct brain regions during perceptual hysteresis, as temporal dependence has been suggested to occur at different levels of visual and cognitive processing, such as perception, attention, decision, and memory (Sacharin et al., 2012; Fritsche et al., 2017; Liaci et al., 2018; Liberman et al., 2018; Witthoft et al., 2018; Mei et al., 2019). For example, the amygdala and the superior temporal sulcus have been suggested to be involved in both emotion processing and perceptual hysteresis (Haxby, Ungerleider, Horwitz, Maisog, Rapoport, & Grady, 1996; Phelps, 2004; Sacharin et al., 2012; Fritsche et al., 2017; Liaci et al., 2018; Liberman et al., 2018; Witthoft et al., 2018; Mei et al., 2019). 

Moreover, psychiatric disorders such as schizophrenia often exhibit belief inflexibility (i.e., difficulties in updating current beliefs, even in the light of disconfirmatory evidence). In the study by Barbalat, Rouault, Bazargani, Shergill, and Blackmore (2012), using facial emotion expressions, patients particularly demonstrated to have belief inflexibility for threatening stimuli. This inflexibility may also be present at the perceptual level in schizophrenic patients, as evidenced by difficulties in the online updating of current sensory perceptual states, even in the light of new sensory evidence. The presence of perceptual inflexibility might contribute to both the patients’ altered perception of reality and the formation of some delusions, as well as to their social cognition deficits. Using an auditory hysteresis paradigm, the study by Martin et al. (2014) showed significantly stronger hysteresis effects in patients than in control subjects, suggesting that perceptual hysteresis may be enhanced in schizophrenia, which might contribute to the experience of delusions in schizophrenic patients. Taken together with our current study, hysteresis paradigms could therefore serve as a way of assessing patients’ altered perceptions of the surrounding environment, providing potentially useful clinical information and giving insights into compromised cognitive processes in neuropsychiatric disorders. 

Conclusions

Our results show that during reality-based, changing emotion expressions recognition, subjects exhibit perceptual hysteresis. We were able to identify one of the two described signatures of the phenomenon: positive hysteresis. These results reinforce the existence of recent temporal context effects on perceptual updates of emotional expressions. Moreover, we found that positive valence content of the stimulus overruled recent perceptual experience. Our findings ultimately suggest that recent perceptual experience and stimulus emotional content may compete in the perceptual decision-making process. Thus, further identification of the underlying neural circuitry is necessary in order to understand how this interaction occurs and ultimately contributes to perceptual hysteresis, in addition to highlighting the role of common brain regions in both emotional processing and perceptual memory. 

Acknowledgments

The authors thank the participants for their participation in this study. The authors also thank Verónica Figueiredo, Alexandre Sayal, João V. Duarte, and Gabriel N. Costa for assistance with data analysis. 

This research work was funded by grants from the Portuguese Foundation for Science and Technology (COMPETE UID/04950/2020, CONECT-BCI POCI-01-0145-FEDER-PTDC/PSI-GER/30852/2017); BIGDATIMAGE project (CENTRO-01-0145-FEDER-000016); and the BIAL project 207/16. 

Commercial relationships: none. 

Corresponding author: Miguel Castelo-Branco. 

Email: [e-mail]. 

Address: Institute of Nuclear Sciences Applied to Health (ICNAS), University of Coimbra, Polo 3, 3000-548 Coimbra, Portugal.