Hysteresis reveals a happiness bias effect in dynamic emotion recognition from ambiguous biological motion

Abstract

Considering the nonlinear dynamic nature of emotion recognition, it is believed to be strongly dependent on temporal context. This can be investigated by resorting to the phenomenon of hysteresis, which features a form of serial dependence, entailed by continuous temporal stimulus trajectories. Under positive hysteresis, the percept remains stable in visual memory (persistence) while in negative hysteresis, it shifts earlier (adaptation) to the opposite interpretation. Here, we asked whether positive or negative hysteresis occurs in emotion recognition of inherently ambiguous biological motion, while testing for the controversial debate of a negative versus positive emotional bias. Participants (n = 22) performed a psychophysical experiment in which they were asked to judge stimulus transitions between two emotions, happiness and sadness, from an actor database, and report perceived emotion across time, from one emotion to the opposite as physical cues were continuously changing. Our results reveal perceptual hysteresis in ambiguous emotion recognition, with positive hysteresis (visual persistence) predominating. However, negative hysteresis (adaptation/fatigue) was also observed in particular in the direction from sadness to happiness. This demonstrates a positive (happiness) bias in emotion recognition in ambiguous biological motion recognition. Finally, the interplay between positive and negative hysteresis suggests an underlying competition between visual persistence and adaptation mechanisms during ambiguous emotion recognition.

Introduction

When recognizing emotions of other people, complex visual and auditory cues can be used to infer emotional labels from facial expressions, body language, or prosody. Accordingly, emotion recognition is often interpreted as a dynamic process whereby perceptual decision often results in nonlinear phase transitions between competing percepts (Kobayashi & Hara, 1993; Liaci et al., 2018; Sacharin, Sander, & Scherer, 2012; Verdade, Castelhano, Sousa, & Castelo-Branco, 2020). Serial dependence effects have been extensively studied in face perception as a mechanism that might be associated with better face recognition abilities, by stabilizing neural representations independently from external noise (Turbett, Palermo, Bell, Burton, & Jeffery, 2019). In the face of the constant changing environments, where noise often compromises perception, it is important to distinguish between useful and noisy information. Integrating through persistence of information helps to improve the signal-to-noise ratio of visual input, leading to sequential effects that help stabilize a neural representation of the object or trait being perceived. While this is essential to mitigate the effects of noise, it is often also important to maximize sensitivity to change. This is the case for negative sequential effects, where adaptation serves as a mechanism, allowing the visual system to process new information. The competition between persistence and adaptation seems to be a characteristic of neural processing, changing depending on the attribute being judged. Taubert, Alais, and Burr (2016) studied the effects of serial dependence in face recognition regarding both gender and expression, revealing that while gender, an attribute that is known not to be susceptible to change, shows strong and consistent positive serial dependencies, expression, which is expected to change over time, presented negative dependencies. However, Liberman, Manassi, and Whitney (2018) registered evidence of a bias of the current perception toward the prior expression only when both faces had similar identities. Face identity itself presents serial dependency effects (Kim & Alais, 2021), as well as several other attributes of face perception, such as attractiveness (Xia, Leib, & Whitney, 2016; Yu & Ying, 2021), trustworthiness, confidence, dominance, intelligence, age, and aggressiveness (Yu & Ying, 2021). These evidences systematically seen in literature (Kim & Alais, 2021; Kok, Taubert, Van der Burg, Rhodes, & Alais, 2017; Liberman et al., 2018; Taubert et al., 2016; Xia et al., 2016; Yu & Ying, 2021) suggest the presence of a general serial dependence mechanism for visual processing regarding social information. 

This phenomenon of serial dependence is closely related to the concept of hysteresis (Figure 1), first described in physics (Warburg, 1881). Classically, positive hysteresis has been described as the maintenance of a certain state of a system despite the changes of input, resulting in delayed transition to the opposed percept when comparing to a control situation where there is no history in the perceptual system (Jiles & Atherton, 1986; Stoner & Wohlfarth, 1997; Warburg, 1881). This positive hysteresis effect reflects perceptual persistence and can be linked with short-term memory mechanisms that maintain the system in the same ongoing state (Kleinschmidt, Büchel, Hutton, Friston, & Frackowiak, 2002; Liaci et al., 2018; Sacharin et al., 2012; Sayal et al., 2020). If the opposite effect occurs, it is named negative hysteresis, possibly caused by fatigue of the current percept, which leads to an anticipation of the perceptual switch earlier on when in comparison with the control situation. Adaptation mechanisms related to fatigue have been proposed to cause negative hysteresis (Assad, Hacohen, & Corey, 1989; Liaci et al., 2018; Lopresti-Goodman, Turvey, & Frank, 2013; Pisarchik, Jaimes-Reátegui, Magallón-García, & Castillo-Morales, 2014; Sayal et al., 2020; Schwiedrzik et al., 2014). Hysteresis can, therefore, be described as the result of a competitive balance between two opposing forces, the stimulus currently being perceived/presented and the competing stimulus representation, resulting in positive or negative serial dependency effects. 

Figure 1.

 

Classical representation of a hysteresis loop as described in physics. The reference curve (black) represents the response of the system if there were no history effect. If there is a tendency of the system to persist in the original state (–1) for a longer period than the reference, we can say there is positive hysteresis (red curve). If the opposite occurs, and the change of state happens prior to the reference moment, there is negative hysteresis (blue curve).

The idea of dimensionality in affective space (Leopold, O'Toole, Vetter, & Blanz, 2001) has been well documented for face perception but is still largely ignored in the field of emotion recognition from biological motion. One reason is the inherent ambiguity of body motion cues in terms of emotion recognition. Recently, a database was developed whereby affect was recognized at levels well above chance and there was a broad range in the ratings of emotional intensity (Ma, Paterson, & Pollick, 2006). Nevertheless, affective cues are ambiguous because they are linked to motion patterns. Understanding emotion recognition and the effects of serial dependence requires forced choices during visualization of ambiguous displays of sequential patterns. 

Hysteresis has been identified previously in human motor patterns, regarding actions such as catching or grasping an object, pointing toward something, or opening drawers, in terms of both action planning and the execution of the movement itself (Adhikari, Quinn, & Dhamala, 2013; Burgess-Limerick, Shemmell, Barry, Carson, & Abernethy, 2001; Kostyukov et al., 2022; Land, Rosenbaum, Seegelke, & Schack, 2013; Lopresti-Goodman et al., 2013; Rostoft, Sigmundsson, Whiting, & Ingvaldsen, 2002; Schütz, Weigelt, & Schack, 2016; Schütz, Weigelt, & Schack, 2017; Valyear, Fitzpatrick, & Dundon, 2019; Weiler & Awiszus, 2000). Recent studies have also demonstrated the presence of hysteresis in facial emotion perception (Liaci et al., 2018; Sacharin et al., 2012; Verdade et al., 2020; Verdade, Sousa, Castelhano, & Castelo-Branco, 2022; Webster, Kaping, Mizokami, & Duhamel, 2004) and auditory stimuli (Martin et al., 2014). The majority of these studies suggested a predominance of positive hysteresis (Liaci et al., 2018; Martin et al., 2014; Sacharin et al., 2012; Verdade et al., 2020; Verdade et al., 2022), with some studies observing negative hysteresis only rarely (Martin et al., 2014; Verdade et al., 2020; Verdade et al., 2022) or observing negative hysteresis in very specific directions (Liaci et al., 2018; Sacharin et al., 2012). 

Face recognition studies are important to understand emotion perception dynamics since face signals are the core of social and emotion cognition. Given the importance of its interpretation to survive and thrive in social interactions, the brain is endowed with specialized face recognition networks (Kanwisher, 2000). Nonetheless, human emotion perception goes much beyond the simple analysis of facial expressions; rather, it is a product of the integration of information of a different nature, including body language, which is expressed through posture and motion patterns (de Gelder & Vroomen, 2000; Meeren, van Heijnsbergen, & de Gelder, 2005; Van den Stock, Righart, & de Gelder, 2007). Despite the fact that partially covering face expressions results in a significant decrease in emotion recognition performance (Carbon, Held, & Schütz, 2022), when the body is also shown, there is a visible impact on the recognition of emotions (Ross & George, 2022). This suggests that body language, and consequently biological motion, strongly affects emotion recognition. The strong evidence of hysteresis in emotion recognition using face expression stimuli (Liaci et al., 2018; Martin et al., 2014; Sacharin et al., 2012; Verdade et al., 2020; Verdade et al., 2022) raises the question whether hysteresis can also reveal perceptual history effects on the perception of emotion from biological motion patterns of human bodies (Duarte, Abreu, & Castelo-Branco, 2022) and whether memory persistence (short-term memory) or adaptation dominates. 

Based on the well-known negativity bias, that is, the tendency humans have to attend to and use preferentially negative information in psychological tasks, especially for socioemotional information (Vaish, Grossmann, & Woodward, 2008), one would expect that it would also be present under the conditions of our experiment. However, prior results suggest that a happiness processing asymmetry may also occur. For example, Kirita and Endo (1995) showed that happy faces can be recognized faster than sad faces when presented in an upright position. Marmolejo-Ramos et al. (2020) found that facial muscular activity alters the recognition of not only facial expressions but also bodily expressions, both appearing as happier when participants smiled. Lee and Kim (2016) showed that emotional valence (happiness) of the stimuli can decrease the threshold for biological motion detection within noise. 

Our research questions were therefore focused on identifying signatures of hysteresis and if they reflected a negative or positive bias (happiness) effect. If the latter case is true, we would expect that this should facilitate the change of percept when the direction of stimulation goes from a negative to a positive percept, as, for example, from sadness to happiness, and persisting on the same percept in the opposite direction. If true, this effect would generalize to the body dynamic expression domain a positive bias in facial expression recognition (Verdade et al., 2020; Verdade et al., 2022). 

Given that both emotion recognition and biological motion perception are severely compromised in neurodevelopmental disorders (Koldewyn, Whitney, & Rivera, 2011; Mason et al., 2021; Pavlova, 2012), this question also has clinical relevance. This emphasizes the relevance of studying emotion recognition from biological motion patterns from the perspective of emotion perception dynamics and the neural mechanisms underlying hysteresis. However, to our knowledge, there are no studies investigating perceptual hysteresis in the recognition of emotions from biological motion. 

Here we study the effect of history by manipulating the dynamic temporal context in the perception of emotion in biological motion patterns. To achieve this goal, we created smooth dynamic transitions between two emotions, happiness and sadness. We hypothesized that there is perceptual hysteresis in biological motion emotion recognition. Furthermore, we asked which of the two neural mechanisms, adaptation and short-term memory, respectively, dominate in this process, resulting in positive and/or negative perceptual hysteresis. 

We found evidence for dominantly positive perceptual hysteresis, with negative hysteresis being also present consistently with a newly identified happiness bias. 

Methods

Participants

Twenty-two healthy participants (mean age of 24.19 ± 3.43 years; 13 females) were recruited. All participants had normal or corrected-to-normal vision and no known history of neurological or psychiatric diseases. Sample size for hysteresis effects has been established in Sayal et al. (2020). All were right-handed except for one, as confirmed by a handedness questionnaire adapted from Oldfield (1971) and available in https://www.brainmapping.org/shared/Edinburgh.php. Participants provided written informed consent prior to the experiment, following protocols approved by the Ethics Committee of the Faculty of Medicine of the University of Coimbra (approval number CE-090/2021), and the study was carried in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). 

Stimuli

Stimuli were created using the Psychophysics Toolbox Version 3 (http://psychtoolbox.org/) (Brainard, 1997) and the Biomotion Toolbox (van Boxtel & Lu, 2013) in MATLAB R2020b (The MathWorks, Natick, MA, USA). 

We used motion-captured data from real people that were then manipulated to create the transitions between emotions as real and smooth as possible. The data used are a part of an online open-access library of motion-captured data for studies of perception of identity, gender, and emotion perception from biological motion (Ma et al., 2006). 

This database comprises several video files with a wide variety of actions combined with different emotions. We selected the data from three actors, one female and two male actors ages 21 to 25 years, performing the action of walking combined with the neutral expression and two emotional states, happy and sad. Each combination of action with emotion was available as a single video instance in the database, in which the actor performed/represented always the same action/emotion during several seconds. The data were captured as point-light displays (Johansson, 1973), composed by the 15 major joints of the human body: head, neck, left and right shoulders, left and right elbows, left and right wrists, pelvis, left and right hips, left and right knees, and left and right ankles. 

We used the neutral state as a reference to create smooth dynamic transitions between the two chosen emotions: happiness and sadness. 

For the morphing of one emotion into the neutral state, we created a trajectory in which each step represented a different graded level of morphing, starting with 100% of the first emotional stage, happy or sad depending on the direction, and decreased the weight of the emotion by 10% while simultaneously increasing the weight of the neutral state by 10%, until reaching 100% of the neutral state and 0% of the initial emotion. The same process was applied from the neutral state until the opposite emotion. 

The happy, sad, and neutral states consisted of a single step of the corresponding original motion-captured data, containing 90 frames and, during 0.9 s each, maintaining the walking velocity of the point-light walker (PLW) independently of the emotion being expressed. We used linear interpolation of the position of each dot to build the intermediate steps with the parametric weights of each state, with the same duration and number of frames as the three original steps. Each step was built by computing all the frames between the last frame of the previous step and the first frame of the next step. By concatenation of two subsequent transitions (happy to neutral + neutral to sad, or sad to neutral + neutral to happy), we obtained two fluid smooth trajectories of 21 steps, starting in one emotion (1 step), morphing (during 9 steps) toward the neutral state (1 step), and subsequently morphing (during another 9 steps) to the opposite emotion (1 step) (see Video S1 of the supplementary material). These stimuli were presented as white dots on a black background (Figure 2). 

Figure 2.

 

Sequence of snapshots from one of the created smooth dynamic transitions. The point-light walkers used were composed of the 15 major joints of the human body: head, neck, left and right shoulders, left and right elbows, left and right wrists, pelvis, left and right hips, left and right knees, and left and right ankles.

To avoid the overlapping of the change of percentage of emotion and the start of another step, the first and last steps were 100% of the emotion at the extreme, and the change of emotion started at the 40th frame of the following step, which resulted in the duration for the first and last steps to be slightly longer. 

For the control conditions, we used individual steps pooled from the morphing trajectories with intervals of the step size of 20%. These steps were then presented in a randomized order in control runs (see Video S2 of the supplementary material). 

Experimental setup

The stimuli were presented using MATLAB R2021b and the Psychophysics Toolbox Version 3, in the center of a liquid-crystal display monitor with a resolution of 1440 × 1080 pixels, a refresh rate of 100 Hz, and a luminance of 18.7 cd/m² in the darker part of the screen and 150.2 cd/m² in the white parts of the screen. Participants were positioned at about 70 cm from the display screen. The movements and the position of the eye and the pupil size were monitored and recorded using Eyelink (SR-Research, Ottawa, Ontario). 

Experimental design

The experiment consisted of a total of three runs of smooth dynamic transitions and three runs of the control conditions (Figure 3). Each run of dynamic transitions (Figure 3A) consisted of three blocks of six trials each. In each block, the trials had a duration of 21.48 s and were interleaved with a black screen for 1.25 s to allow the participant to rest before the next trial. At the beginning of each run as well as at the end of each block and the end of the run, a fixation cross was presented for 10 s. In total, each of these runs had a duration of 9.50 min. Participants were instructed to indicate the perceived emotion initially and the moments of perceptual switch, both to the neutral state and to the opposite emotion by pressing and holding one of three keyboard keys, one for each state—happy, sad, or neutral. 

Figure 3.

 

(A) Example of the organization of a dynamic run with a total duration of 9.50 min consisting of three blocks with six trials each. Participants performed three of these runs. (B) Example of the organization of a control run, with a total duration of 11.11 min consisting of four blocks with 33 trials each. Participants performed three of these runs.

The control runs (Figure 3B) consisted of four blocks with 33 trials. Each trial contained one step with a duration of 0.9 s, followed by a black screen for 4 s, resulting in a total duration of 4.9 s per trial and 161.7 s per block. At the beginning of each run as well as at the end of each block and run, a fixation cross was presented for 5 s. In total, each of these runs had a duration of 11.11 min. The participants were instructed to identify the emotion presented in each trial by pressing one of the three buttons after the presentation of the stimulus, to indicate the perception of the step as happy, neutral, or sad. 

Preanalysis data processing

The first step of the analysis was to divide and classify every trial, according to the responses given during the experiment. Only the trials classified as completed trajectories—that is, trials where the participant identified the three emotional states in the correct order (E1-N-E2)—were used in subsequent analyses as these were the only trajectories including a complete sequence of perceptual switches and that started and ended in the emotions corresponding to the stimuli. 

Based on the design of the stimulus, the frames were parametrized in terms of the percentage of the positive emotion depicted by the corresponding step, being –100% absolute sadness (original), 0% neutral (original), and 100% absolute happiness (original). For example, the frames of the third step of a dynamic trajectory initiating in happy are labeled with 80%. 

Using the frame numbers in which the participant reported the changes of perception and the corresponding parametrization of the percentage of emotions, we attributed a percentage of emotion to each report. 

Tracing the control curves

Emotion perception under control conditions (absence of history) presents a nonlinear (sigmoidal) behavior for both emotions, sadness and happiness. After the perception of the neutral state, the point of transition is reached, which is estimated to be different from participant to participant. The neutral perception is briefly held before the change to the other percept. 

The neutral point is the point where the control curve crosses with y = 0, with the y-axis being the perceptual outcome, with –1 being sadness, 0 the neutral state, and 1 happiness. Theoretically, this point should be located at x = 0, with the x-axis the axis of the parametrization of the positivity of the emotion in the stimuli and 0 the point where the videos presented are exactly the same as the original videos where the actors acted as being in a neutral state. 

To obtain the control curves, we first attributed to each report the value of –1 if the response was sad, 0 if neutral, and 1 if happy and then calculated the mean of the reports for each corresponding percentage of emotion displayed in the control stimuli. Since the stimuli were created based on the motion data captured from three different actors, we traced the control curves for the three actors separately to assess possible differences perceived by the participants. 

Defining hysteresis metrics

Classically, we can measure hysteresis using two different metrics: the spacing between the curves of each trajectory or the distance of the two inflection points, that is, the moment of the change of perception. Using both these metrics, we established three criteria for the existence of hysteresis, which are depicted in Figure 4. These are as follows: Criterion 1, the distance between the x values of the trajectory and the control curve of the corresponding actor in which y = 0 is different from zero; Criterion 2, the distance in y between the trajectory and the control curve of the actor for the two previous points and the two next points of the x value in which y = 0 is different from zero and is always positive or always negative, meaning the two trajectories do not cross during this period; and Criterion 3, considering the trajectory only until the x value for which y = 0, the area between the trajectory and the curve drawn using the confidence interval of the points of the control curve is bigger than the area between this curve and the control curve. 

Figure 4.

 

Example of a “happy to sad” trajectory in blue and the control curve for the corresponding participant in gray, with the standard error in each sampling point of the curve and the visual representation of the three mathematical criteria defined for the classification of the trajectories regarding the presence of hysteresis and the type of hysteresis. On the y-axis, –1 represents sadness, 0 the neutral state, and 1 happiness, while on the x-axis, –100% is sadness, 0 neutral, and 100% happiness.

For any given trajectory to present hysteresis, it needs to fulfill at least two of the three criteria; otherwise, it is considered a null trajectory (N). Then, trajectories with hysteresis can be distinguished by two types: the ones showing positive hysteresis (PH), or persistence, in which the participant holds the current perception for longer, causing a positive lag relative to the control, and the ones showing negative hysteresis (NH), based on an adaptation mechanism, in which there is an early perceptual switch, causing a negative lag. We also considered a third category, undefined (UN), to encompass the trajectories where the behavior of the curve initially seems to follow a pattern of hysteresis, but it abruptly changes and starts to follow the control curve. Figure 5 presents examples of these four possible classifications. 

Figure 5.

 

Examples of classified trials according to the defined criteria: (A) positive hysteresis, characterized by a positive lag in the perception; (B) negative hysteresis, in which there is a negative lag in the perception; (C) null, corresponding to the trajectories without hysteresis; and (D) undefined, which groups the trajectories with a behavior different from what we expected for either positive or negative hysteresis. On the y-axis, –1 represents sadness, 0 the neutral state, and 1 happiness, while on the x-axis, –100% is sadness, 0 neutral, and 100% happiness.

Furthermore, we added a visual expert review based on visual classification, which was performed by two independent raters individually at first, who then discussed the cases of disagreement until reaching consensus. The visual expert classification always prevailed over classification derived from the mathematical criteria whenever they were not concordant. 

Results

Group-level analysis

Control curves

Comparing the three curves, it is clear that there is a different level of ambiguity for each actor with the biological motion patterns from Actor 1 being more ambiguous than the other two. This is an extreme consequence of affective cue ambiguity (because they are based on motion/posture features). Accordingly, the average perceptual control curve for the hysteresis curves in Actor 1 shows evidence for perceptual uncertainty, even under alternative force choice conditions (Figure 6). However, when we introduce serial dependence in perceptual decision, this ambiguity that is typical of biological motion is dramatically reduced (the hysteresis curves). A higher degree of ambiguity in the perception of the emotion of an actor translated itself into a close to linear control curve instead of the expected sigmoid. This ambiguity can be explained by several different factors such as the fact we are using real motion-captured data performed by amateur actors, which might have led to variable representations of the emotion, and the actors themselves might have had different levels of expression regarding these emotions as each human being is unique in expressing their emotions. Furthermore, perceiving and expressing the emotions through motion cues might be more difficult than for facial expressions. Emotion perception is an individual-specific process affected by a number of factors that makes it unique for each of us, and therefore, even watching the exact same videos, the neutral point changes across participants. 

Figure 6.

 

Group averages of each direction, with red being the direction “sad to happy” and blue the direction “happy to sad,” and the control curve, in gray, for each actor. The shaded areas represent the confidence interval of 95%. On the y-axis, the perceptual outcome, with 1 being happiness, 0 the neutral state, and –1 sadness, is represented. The x-axis is parametrized according to the positivity of the emotion present on the videos, being –100% sadness and 100% happiness.

Dynamic transitions

In general, even for Actor 1, which depicted more ambiguous emotional expressions, the participants were able to easily identify the emotions through the biological motion stimuli and to see the transitions between the three emotion types without major difficulty. 

We traced the mean trajectory for each direction (happy–neutral–sad and sad–neutral–happy) and each subject and the mean of those to achieve the mean trajectory of the group for each direction (Figure 6). This was calculated separately for the three actors given the differences found while studying the control curves. 

For all three actors, there is a clear spacing between the trajectories of both directions, suggesting they are entirely independent from each other, thus presenting a clear effect of the history on the perception of the current stimulus (e.g., hysteresis). The observation that the path of perception from “happy to sad” is not the same as the path from “sad to happy” is indeed a signature of perceptual hysteresis. 

The control curves can be used as a reference to the trajectory without the added effect of hysteresis, allowing for the classification of the type of hysteresis at the group level, at least for Actors 2 and 3, since in these cases, the control curve shows the expected behavior (Figure 6). Although Actor 1 also shows evidence for hysteresis, he presents an unusual control curve that does not allow for the classification of the type of hysteresis in either direction. For this reason, Actor 1 was excluded from the subsequent analysis of the results related to identify the type of hysteresis (positive or negative). However, not being able to classify the type of hysteresis in terms of sign does not impede acknowledging its existence, as it is clear that both directions follow different and well-separated paths, showing a clear effect of the history on the trajectory as well. In fact, the diagnosis of the presence of hysteresis does not require the presence of a control curve. 

Inferential statistical analysis

To better analyze the existence of perceptual hysteresis, we proceeded to study the moment of the perceptual switch from the first emotion to the neutral state as well as the area between the curves of each direction and the control and between both directions, as these two factors are often mentioned in the literature as measures of the hysteresis effect (Sayal et al., 2020; van Rooij, Atmanspacher, & Kornmeier, 2016). 

Shapiro–Wilk tests revealed that not all the data followed normal distributions; therefore, we performed nonparametric Wilcoxon signed-rank tests to assess statistically significant differences between the two directions as well as each direction and the control. 

Comparing the perceptual switch moment of all three actors for both directions, “happy to sad” and “sad to happy,” the Wilcoxon signed-rank revealed significant differences (Table 1). We found significant differences in the moment of perceptual switch to the neutral state in each direction compared with the control for both Actor 2 and Actor 3, as well (Table 1), suggesting that the history does affect the perception of the current stimulus. 

Table 1.

 

Wilcoxon signed-rank test results for the distribution of the percentage of the perceptual switch moments for each direction and control and for both directions.

 

Subject-level analysis

Classifications of the individual trajectories

Table 2 presents the results of the classifications using the criteria previously defined as well as after visual review. 

Table 2.

 

Summary of the classification of each individual trajectory, based on the mathematical criteria and after visual revision. Notes: The visual classification prevailed over the classification with the mathematical criteria. PH/NH = positive/negative hysteresis; UN = undefined.

 

The two raters were trained using the examples of hysteresis curves well known from physics as well as examples of the four possible classifications. The comparison between both classifications resulted in only a total of 55 cases of disagreement before consensus discussion, out of a total of 685 trajectories (Table 2), which results in a large Cohen's kappa of 0.88. 

Although positive hysteresis seems to be overall predominant (348 out of 685 trajectories), the direction “sad to happy” yields a proportion closer to an equilibrium between negative and positive hysteresis, with even more cases of negative hysteresis than positive hysteresis for Actor 2 (Table 2). These results are in accordance with the graphs of the group average where we could observe a prevalence of positive hysteresis in both actors, with Actor 2 having the direction “sad to happy” closer to the control curve in the second part of the trajectory (Figure 6). 

The distribution of both types of hysteresis per subject shows hidden heterogeneity that was not apparent in the overall distributional analysis. We observe, however, that while positive hysteresis seems to be consistently predominant in the direction “happy to sad” (the “happy” percept persisting), there are frequently cases where negative hysteresis is predominant in the direction “sad to happy” (the “happy” percept being an “attractor”) for both actors, which is in accordance with an “happiness bias” effect. 

To evaluate if the type of hysteresis, negative or positive, depends on the abovementioned directions, we performed a chi-square test. Compiling the results from both actors, there is a dependence of both variables studied (χ²(1, N = 507) = 14.713, p < 0.001); therefore, the type of hysteresis depends on the direction of the trajectory between emotions, which clearly manifests an effect of the history in the trajectory. 

For each direction, the trajectories were separated accordingly to the previous classification into two different distributions, one with negative hysteresis and other with positive hysteresis. To analyze the differences between these distributions, we used two parameters: the perceptual switch moment for both classifications and the area between the two curves being compared. Shapiro–Wilk tests revealed that the perceptual switch moments and the areas between the curves did not follow a normal distribution. 

The results from Mann–Whitney tests for the perceptual switch moments comparison reveal significant differences between both types of hysteresis in both directions for the two actors (Table 3) as well as significant differences between the areas under trajectories of both types of hysteresis for both directions, “happy to sad” and “sad to happy,” for the two actors (Table 3). 

Table 3.

 

Mann–Whitney test results for comparison between the two types of hysteresis for the distribution of the percentage of the perceptual switch moment of the trajectory and the point of neutral state in the control curve and for the distribution of the areas between the control curve and the trajectory.

 

Discussion

The present study was conducted to investigate the effect of the dynamic temporal context, as manifested by hysteresis signatures, in the perceptual recognition of emotions from biological motion, which is inherently ambiguous, even under forced-choice paradigms. Hysteresis paradigms allow one to impose trajectories in affective space. The existence of “face affective spaces” or alike (Leopold et al., 2001) allows one to implement stimulus trajectories across particular directions in these spaces and represent a special form of study of sequential effects in visual emotion recognition. We generated smooth dynamic transitions between two emotion categories, happiness and sadness, and specifically asked if persistence or adaptation mechanisms dominated (positive vs. negative hysteresis, respectively) and if a negative or a positive emotional bias would be observed, the former being more frequently reported in other domains. 

Our research questions were therefore deeply anchored on the notion that biological motion perception is heavily influenced by our own experience in daily life social and emotional interactions. As expected, this perceptual ambiguity translated into the number of completed trajectories varying across participants as well as the neutral point also varying between participants. The fact that affective motion/pose cues are inherently ambiguous explains why the average perceptual control curve for the hysteresis curves in Actor 1 was flat. However, the introduction of serial dependence in perceptual decisions led to much less ambiguous hysteresis curves. In the database we used (Ma et al., 2006), affect was recognized at levels well above chance, but ambiguity was recognized as an intrinsic feature. Here we show that such ambiguity is dampened by serial dependence. 

Regarding the first research question, the perception of emotions on the dynamic transitions was significantly different from that on the control curves traced individually for each participant, favoring visual persistence mechanisms and therefore showing evidence for positive perceptual hysteresis. 

The fact that positive hysteresis is predominant (Figure 6) extends to the body emotion recognition domain previous observations concerning facial expressions (Liaci et al., 2018; Sacharin et al., 2012; Verdade et al., 2020). We could, however, identify that at the individual level, although positive hysteresis is still predominant in general, we could determine instances of negative hysteresis (Table 2), which in some participants was even predominant, largely due to a happiness bias effect (see below). 

The positive lag in the emotion perception observed in the trajectories with positive hysteresis can be explained by short-term memory visual persistence mechanisms that help in stabilizing a percept, reducing ambiguity, and therefore resulting in its maintenance in visual awareness. 

Concerning the second research question, we found evidence favoring a positive (happiness) bias effect, which contradicts a mainstream prediction of a negative bias for socioemotional cognition (Vaish et al., 2008). This is in accordance with our previous finding of a positive (happiness) bias regarding temporal context in the recognition of emotions from facial expressions (Verdade et al., 2020; Verdade et al., 2022). This suggests that there may be some processing advantages for happiness stimuli that are consistent with the notion that emotional valence (happiness) of the stimuli can facilitate the processing of biological motion (threshold within noise) or that happy faces are recognized faster than sad faces when presented in an upright position (Kirita & Endo, 1995; Lee & Kim, 2016). 

The observation of persistence, or positive hysteresis, in trajectories where the first emotion is happiness is quite remarkable. Indeed, we did observe a statistically significant prevalence of positive hysteresis, in the direction “happy to sad,” although a relative proportion of about one third of the trajectories in this direction present negative hysteresis (Table 1). Negative hysteresis is usually attributed to adaptation mechanisms that favor the switch to the other percept earlier than the physical transition occurs (Assad et al., 1989; Liaci et al., 2018; Lopresti-Goodman et al., 2013; Pisarchik et al., 2014; Sayal et al., 2020; Schwiedrzik et al., 2014). Interestingly, in the “sad to happy” direction, although positive hysteresis is still predominant, the presence is much weaker, in line with a happiness bias serving as an “attractor.” In Actor 2, negative hysteresis even surpasses positive hysteresis. This is substantiated by the fact that a statistically significant difference in the proportion of cases of positive and negative hysteresis occurred as a function of the direction of the trajectories. This equilibrium of the cases seems to be a result from the effect of the positive (happiness) bias that favors the earlier perceptual switch to the happy percept when starting in sadness. 

Our previous work with emotion recognition from facial expressions (Verdade et al., 2020; Verdade et al., 2022) showed clear evidence of the presence of hysteresis in dynamic trajectories between three pairs of emotions: happiness and sadness, happiness and anger, and anger and sadness. Using facial expressions, we could also demonstrate the presence of a happiness bias. Accordingly, the perception of happiness preferentially persisted in trajectories from happiness to sadness or to anger. Moreover, the opposite trajectories had low persistence of negative emotions, because they converged rather swiftly to happiness percepts. Our findings generalize to the body emotion perception domain our previous findings concerning emotion recognition from facial expressions (Verdade et al., 2020; Verdade et al., 2022). However, we also observed the presence of negative hysteresis in both directions when using biological motion patterns. 

Recently, Ross and George (2022) studied the effect of the use of masks on the recognition of emotions and concluded that when the body was shown, there was not a big impact in performance deterioration, which suggests that biological motion perception plays an important role in the recognition of emotional states in the absence of facial expression information. 

The presence of trajectories with negative and positive hysteresis in the same direction in some participants reveals a possible competition between the two neural mechanisms, short-term memory and adaptation, with dominance of the first, which is in accordance with the hypothesis that regarding socioemotional cognition, short-term memory mostly overrules adaptation. In the future, neurophysiological studies (e.g., functional magnetic resonance imaging experiments) may help determine the neural correlates of such competition. 

The phenomenon that we describe here is quite distinct from the negativity bias, characterized by the interpretation of ambiguous stimuli as negative, that is often observed in depressed patients with depression (Ito et al., 2017). Our findings also depart from the negative attentional bias that is reflected in attentional maintenance indices (i.e., first fixation duration and total fixation time) for sad faces in patients with major depressive disorder (Duque & Vázquez, 2015) and suggests that the happiness bias observed here may be reversed in diseases of mood. This question can potentially be addressed in future studies. If true, measures of bias might actually be used as indexes of change in psychological well-being. 

As concluding remarks, to our knowledge, we demonstrate for the first time that positive hysteresis and a positive (happiness) bias dominate in emotion recognition from biological motion patterns. We confirmed the existence of hysteresis in emotion recognition using biological motion patterns while showing that the happiness bias seen in facial expressions is a transversal phenomenon also seen in emotion recognition from biological motion patterns. Notably, it is widely known that the perception of both emotional and biological motion stimuli is severely compromised in neurodevelopmental disorders (Koldewyn et al., 2011; Mason et al., 2021; Pavlova, 2012), which leads to the question of whether the underlying neural mechanisms are also compromised in such disorders. Understanding the behavioral underpinnings of emotion recognition and perceptual decision-making in this context is the first step in the understanding of alterations in perceptual states and its neural correlates. 

Acknowledgments

The authors thank the participants for their involvement in this study. We are also very grateful to Isabela Chedid for help in reviewing and classifying hysteresis in individual trajectories. 

Supported by grants funded by Fundação para a Ciência e Tecnologia, UIDB/4950/2020, UIDP/4950/2020, DSAIPA/DS/0041/2020 | COVDATA | IN1111, and PTDC/PSI-GER/1326/2020. FCT also funded an individual contract to JVD (CEECIND/00581/2017). 

Open practices statement: The datasets generated during the current study are available from the corresponding author on reasonable request. 

Commercial relationships: none. 

Corresponding author: Miguel Castelo-Branco. 

Email: [e-mail]. 

Address: Institute of Nuclear Sciences Applied to Health (ICNAS), University of Coimbra, Portugal University of Coimbra, Polo 3, Coimbra 3000-548, Portugal.