Adaptation reveals that facial expression averaging occurs during rapid serial presentation

Abstract

How do we interpret the rapidly changing visual stimuli we encounter? How does our past visual experience shape our perception? Recent work has suggested that our visual system is able to interpret multiple faces presented temporally via integration or ensemble coding. Visual adaptation is widely used to probe such short term plasticity. Here we use an adaptation paradigm to investigate whether integration or averaging of emotional faces occurs during a rapid serial visual presentation (RSVP). In four experiments, we tested whether the RSVP of distinct emotional faces could induce adaptation aftereffects and whether these aftereffects were of similar magnitudes as their statistically averaged face. Experiment 1 showed that the RSVP faces could generate significant facial expression aftereffects (FEAs) across happy and sad emotions. Experiment 2 revealed that the magnitudes of the FEAs from RSVP faces and their paired average faces were comparable and significantly correlated. Experiment 3 showed that the FEAs depended on the mean emotion of the face stream, regardless of variations in emotion or the temporal frequency of the stream. Experiment 4 further indicated that the emotion of the average face of the stream, but not the emotion of individual faces matched for identity to the test faces, determined the FEAs. Together, our results suggest that the visual system interprets rapidly presented faces by ensemble coding, and thus implies the formation of a facial expression norm in face space.

Introduction

How do we interpret the rapidly changing visual world? Specifically, how do we interpret a stream of different faces? Potter and colleagues (e.g., 1975, 1976, 2014) found that subjects were able to detect the target from a sequence of distractors embedded in high speed visual streams, with such a visually presented sequence of items known as rapid serial visual presentation (RSVP). This ability in detecting a target from a sequence of stimuli is weakened as the temporal frequency of the sequence increases. When we are exposed to a large amount of information, our perception may be narrowed down to a finite number of objects that are presented, possibly due to the temporal limitation and capacity of working memory (Brady & Alvarez, 2015; Haberman & Whitney, 2012; Nieuwenstein & Potter, 2006). Alternatively, we may process such a mass of information by obtaining the averaged “gist” in ensemble statistics (Alvarez & Oliva, 2009; Haberman & Whitney, 2012), with the term ensemble statistics used interchangeably with “ensemble representation,” “statistical summary,” “ensemble coding,” “ensemble perception,” and “summary representation” (Alvarez, 2011; Haberman & Whitney, 2007, 2012). Summarizing the similarities of large quantities of stimuli statistically has been found in low-level features like orientation and direction, through to high-level information such as facial expression, identity and gaze viewpoint (Alvarez, 2011; Alvarez & Oliva, 2008, 2009; Ariely, 2001; Haberman, Brady, & Alvarez, 2015; Haberman & Whitney, 2007, 2009, 2012; Leib et al., 2014; Sweeny & Whitney, 2014). Haberman and colleagues (2009) showed that participants perceived the average expression in a temporal sequence of different faces in the same way that they perceived a single face presented repeatedly, thus suggesting a form of facial averaging across the different faces. Research has shown that this statistical summary can occur spatially (Alvarez & Oliva, 2008; Ariely, 2001; Chong & Treisman, 2003; Haberman et al., 2015; Leib et al., 2014; and is a possible mechanism for visual crowding, Fischer & Whitney, 2011) and temporally (an occurrence relatively less explored, Haberman, Harp, & Whitney, 2009). 

However, such ensemble coding is only one hypothesized way in which sequentially presented stimuli are processed. For example, visual masking can occur when presenting a target stimulus in the temporal vicinity of other stimuli, and causes the “target” or “test” stimulus to be judged less accurately in comparison to when judged without any masking. There are two major theories to explain visual masking: “interruption,” and “competition” (Keysers & Perrett, 2002; Kahneman, 1968). The interruption theory suggests that the processing of the first stimulus is interrupted in favor of the new stimulus when the latter is presented. This interruption leaves the processing of the first stimulus unfinished, thereby impairing its perception. Following this theory, the last stimulus in the RSVP sequence will be the strongest and of the largest influence. The second theory, competition theory, is a modified version of the first theory but includes an explanation at the neuronal level: There is competition in neural processing between the two stimuli, but the new stimulus generally wins because the neural responses to stationary patterns (the first stimulus) decrease over time. When the presentation gap between consecutive stimuli is shortened, or the temporal frequency of a visual stimulus is more rapid, the neural activities may fuse together. Following this theory, the last stimulus in the RSVP sequence would still be the strongest or of largest influence. In our current study, we will test which of the above theories (ensemble coding, interruption, or competition) will best explain emotion adaptation across an RSVP paradigm. 

Here, we use a visual adaptation paradigm to investigate the perception of an RSVP of emotional faces. When viewing a happy face for a few seconds, subsequently presented faces are perceived as less happy. These adaptation aftereffects are believed to arise due to neuronal populations specialized for detecting the adapting stimulus's characteristics (e.g., facial happiness) becoming habituated during the adaptation phase, thus explaining why adaptation has been called “the microelectrode for psychologists” (Frisby, 1979). Mechanisms of face adaptation have been proposed from different perspectives, neurally, psychologically, and computationally, such as norm-based coding (Burton, Jeffery, Calder, & Rhodes, 2015; Susilo, McKone, & Edwards, 2010) and exemplar-based, multichannel coding (Calder, Jenkins, Cassel, & Clifford, 2008; Lawson, Clifford, & Calder, 2011). Although the evidence for them came from different aspects of face processing (e.g., expression, identity, and gender adaptation for norm-based coding; and direction of eye gaze, head orientation, and face viewpoint for multichannel coding; Fang & He, 2005; Seyama & Nagayama, 2006), both models suggest that the consequence of adaptation is normalization of the responses and recalibration for the mean stimulus level (Webster, 2014, 2015). Therefore, the process of adaptation allows the visual system to recalibrate based on changing surroundings. This plasticity is ubiquitous and occurs in multiple stages of visual processing, such as in curve, face, object, scene, and motion perception (Campbell & Burke, 2009; Fox & Barton, 2007; Gibson 1933; Gibson & Radner, 1937; Greene & Oliva, 2010; Hsu & Young, 2004; Köhler & Wallach, 1944; Rhodes & Jeffery, 2006; Rhodes, Jeffery, Watson, Clifford, & Nakayama, 2003; Verstraten, 1996; Webster, Kaping, Mizokami, & Duhamel, 2004; Webster & MacLeod, 2011; Wohlgemuth, 1911). Adaptation has been demonstrated many times by individual items, such as a single grating or face. Yet people see many objects at once, both in space and across time, sometimes very quickly. Is the process of adaptation inclusive of many objects seen over time? Do several faces seen in quick succession influence adaptive coding in the same way as a single face? And if so, how might this rapid integration of faces occur? 

In the current study, we examined the perception of emotion of sequentially presented faces by adapting participants to the RSVP of emotional faces. We found that adapting to faces during an RSVP sequence generated similar facial expression aftereffects as adapting to the morphed average face of the RSVP face sequence, regardless of the variations of emotions and temporal frequency in the RSVP sequence. When half of the faces in the RSVP sequence were happy and half were sad, any aftereffects were abolished. We also found that the aftereffect was determined by the overall averaged facial emotion in the RSVP face sequence, instead of being disproportionately influenced by individual faces matched for identity to the test faces. Through the use of visual adaptation, we have provided new evidence to suggest that the temporal ensemble coding of emotional faces occurs during an RSVP sequence. Our findings have clear psychophysical, neural and computational implications. 

Experiment 1

Faces presented in a rapidly changing sequence can either be segregated for target detection or grouped together. In the current study, we tested whether passively viewing consecutively presented emotional faces would generate facial expression aftereffects. We adapted the participants to a sequence of faces, and asked them to judge subsequently presented faces' emotions. The adapting faces were from different identities, but with the same emotion (happy or sad). If the subjects did group the RSVP sequence of emotional faces together during adaptation, we would expect to see a facial expression adaptation aftereffect. 

Methods

Observers

Ten subjects (five males, five females; mean age 22.8 years), with normal or corrected-to-normal vision, participated in Experiment 1. One of the subjects (HY) was the experimenter, and the other nine subjects were naïve to the purpose of the experiment. All subjects gave written consent before testing. This study was approved by the Internal Review Board (IRB) at Nanyang Technological University, Singapore, in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving human subjects. 

Apparatus

Visual stimuli were presented on to a 17-in. Philips CRT monitor (refresh rate 85 Hz, spatial resolution 1024 × 768 pixels). The monitor was controlled by an iMac Intel Core i3 computer running Matlab R2010a (Mathworks, Natick, MA) via Psychophysics Toolbox extensions (Brainard, 1997; Pelli, 1997). Subjects were seated in the dimly lit room with their heads rested on the chin rest placed at a distance of 75 cm in front of the monitor. Each pixel subtended 0.024° on the screen. 

Stimuli

All the face stimuli were chosen from Karolinska Directed Emotional Faces (KDEF; Lundqvist, Flykt, & Öhman, 1998) database. Faces with different emotions (happy, sad, and neutral) from the same identity were selected. We used the Psychomorph software (DeBruine & Tiddeman, 2015) to manipulate and morph these faces. 

Adapting stimuli:

The adapting stimuli were the original faces from KDEF face set. Faces of 10 different identities were chosen, each identity with happy (AF01HAS, AF05HAS, AF07HAS, AF17HAS, AF19HAS, AF20HAS, AF22HAS, AF26HAS, AF29HAS, and AF34HAS) and sad (AF03SAS, AF06SAS, AF07SAS, AF11SAS, AF13SAS, AF20SAS, AF26SAS, AF29SAS, AF32SAS, and AF34SAS) emotions (Figure 1A and B). We used the happy and sad adapting stimuli because these emotions have been widely used in facial expression aftereffect studies (for example, Fox & Barton, 2007; Hsu & Young, 2004), and are believed to induce contrastive adaptation aftereffects (Xu, Dayan, Lipkin, & Qian, 2008). Only the face region of each face image was displayed in the experiment. All adapting stimuli were of a size of 2.40° × 3.02°. 

Figure 1

 

Stimuli used in Experiment 1. (A) Adapting faces, ten happy faces from different identities used as adapting stimuli (AF01HAS, AF05HAS, AF07HAS, AF17HAS, AF19HAS, AF20HAS, AF22HAS, AF26HAS, AF29HAS, and AF34HAS). (B) Adapting faces, ten sad faces from different identities used as adapting stimuli (AF03SAS, AF06SAS, AF07SAS, AF11SAS, AF13SAS, AF20SAS, AF26SAS, AF29SAS, AF32SAS, and AF34SAS). (C) Test faces, ten morphed faces used as test stimuli.

Test stimuli:

We averaged all of the 35 neutral female faces of different identities from the KDEF face database to create a neutral face template. Similarly, we averaged 35 happy faces of these identities to generate the happy face template. We then morphed the neutral and happy face templates to create a sequence of test faces, varying in proportion of happiness. The proportions of happiness in test images were equal to 0%, 10%, 20%, 30%, 35%, 40%, 45%, 50%, 60%, and 70% (Figure 1C). Only the face region of test stimuli is displayed in the experiment. All test stimuli were of a size of 2.40° × 3.02°. 

Procedure

We measured the facial expression aftereffect by adapting participants to the RSVP face sequence with different identities but the same emotion. The general procedure was adapted from previous studies in facial emotion aftereffects (i.e., Xu et al., 2008). The adapting stimulus was either the happy or the sad RSVP face sequence (examples in Figure 1A and B). The test stimulus was one of the morphed faces (Figure 1C). The face stimuli were horizontally aligned with the fixation cross, with a center-to-center distance of 3.5°, presented randomly to the left or right of the fixation point. There were three conditions in total: happy RSVP adaptation, sad RSVP adaptation, and a baseline condition. We used the method of constant stimuli and the one interval, yes-no design in all experiments. These three conditions were run in separate blocks. Within each block, each test face was repeated 10 times, in a random order. Each block lasted around 10 minutes, and there was a 10-min rest in between two consecutive sessions, to avoid carryover effects to the next block. The order of the blocks was random for each subject. Each subject went through all of the blocks on the same day. The whole experiment lasted around 50 minutes. Data collection for each block started after subjects had sufficient practice trials (typically 10–20) to feel comfortable with the task. 

Subjects started each block of trials by fixating at the central cross and then pressing the space bar (Figure 2). After 1494 ms (127 frames), for each adaptation block the adapting RSVP face sequence appeared for 3764 ms (320 frames). In the adapting streams, each image lasted 23.5 ms (two frames, temporal frequency at 42.5 Hz) on the screen and then was replaced by another picture at the same location with no interval. There were 10 happy or sad faces of different identities in total, each repeated 16 times in random order in the RSVP sequence. After a 506 ms (43 frames) interstimulus interval, a test stimulus appeared for 400 ms (34 frames), masked by two 47 ms (four frames) random Gaussian noise masks. The short test stimulus duration was selected to enhance aftereffects (Wolfe, 1984). The mask was displayed to reduce the effect of the afterimage. For the baseline blocks without adaptation, only a test stimulus was shown in each trial for 400 ms (34 frames). A 50 ms beep was then played to remind subjects to report their perception of the test stimulus. Subjects had to press the “H” or “N” key to report happy or not happy. After the response, the next trial began. Subjects received no feedback on their performances at any time. 

Figure 2

 

Trial Sequence of the happy RSVP adaptation condition. Subjects fixated on the cross and pressed the space bar to initiate a trial. After 1494 ms, the adapting RSVP face sequence appeared for 3764 ms. After a 506 ms interstimulus interval (ISI), a test face appeared for 400 ms, masked by two 47 ms random noise masks. The screen position of the adapting RSVP face sequence was identical to that of the test faces. A beep was then played to remind the subjects to report the perceived expression of the test face. Subjects had to press either the “H” or the “N” key to indicate a perception of a happy or not happy expression. Experimental parameters for all conditions and experiments are detailed in the Methods section.

Data analysis

The data were sorted into fractions of happy responses to each test stimulus per adaptation condition. The test stimuli were parameterized according to the proportion of happiness in the morphed test faces. The fractions of happy responses were then plotted against the test faces, and the resulting psychometric curves were fitted with a sigmoidal function f(x) = 1/[1 + e^{−a(x − b)}], where a/4 is the slope and b gives the test-stimulus parameter corresponding to the 50% point of the psychometric function, the point of subjective equality (PSE). Adaptation aftereffects were measured as the difference between the PSEs of the adapting condition relative to its baseline condition. We used a two-tailed paired t test to compare subjects' PSEs for different conditions in an experiment. 

Results

The results from a naive subject judging the facial expression of the test faces under various conditions are shown in Figure 3A. We plotted the fraction of happy responses as a function of the proportion of happiness of the test faces. The black psychometric curve is the baseline condition without adaptation. After adapting to the happy RSVP face stream, the subject perceived happy expressions less frequently, and the psychometric curve (blue, happy RSVP) shifted to the right. After adapting to the sad RSVP face stream, the subject perceived happy expressions more frequently, and the psychometric curve (red, sad RSVP) shifted to the left. This is the standard facial-expression aftereffect (Hsu & Young, 2004; Webster et al., 2004, Webster & MacLeod, 2011). The new finding here is that after adapting to the RSVP face stream, which contains multiple faces from different identities but the same emotion (happy or sad), the subject is able to extract the emotion from the face stream during adaptation, and perceive the opposite expressions more frequently in the test faces. 

Figure 3

 

The effect of RSVP face stream adaptation (Experiment 1). (A) Psychometric functions from a naive subject under the following conditions: Baseline, No adaptation baseline (black circle, solid line); happy RSVP, adaptation to the happy RSVP face stream (blue filled diamond, dashed line); sad RSVP, adaptation to the sad RSVP Face stream (red open diamond, dashed line). For each condition, the happy responses were plotted as a function of the proportion of happiness of the test face. (B) Summary of all ten subjects' data. For each condition, the average PSE relative to the baseline condition and the standard error of the mean (SEM) were plotted. The p value shown for each condition in the Figure was calculated against the baseline condition using the two-tailed, paired t test.

To quantify the aftereffects and summarize the results from all ten subjects, we determined the PSE (the proportion of happiness corresponding to 50% happy responses) for each psychometric curve of each subject. Figure 3B shows the mean PSEs relative to the baseline condition. A positive value means a rightward shift of the psychometric curve (less happy responses) relative to the baseline. A negative value means a leftward shift of the psychometric curve (more happy responses) relative to the baseline. The error bars indicate SEMs. The p value for each adaptation condition in the figure was obtained by comparing the PSE shift of each condition against that of the baseline condition via a two-tailed paired t test. Both aftereffects were significant, with a positive aftereffect for happy RSVP adaptation condition, M = 0.156, SEM = 0.0251; t(9) = 6.219, p < 0.01, and a negative aftereffect for sad RSVP adaptation condition, M = −0.141, SEM = 0.0249; t(9) = −5.669, p < 0.01. 

Discussion

We found that adapting to rapidly changing sequences (RSVP) of faces with the same emotion (happy or sad) generated significant facial expression aftereffects. This is interesting as in each RSVP stream there were 160 faces from 10 different identities with the same emotion, with each face presented for only 23.5 ms. Although the subjects were not required to identify individual faces during the adaptation period, they were still able to extract the happy or sad emotional information in the RSVP stream during passive viewing. The prolonged exposure to the emotion in the RSVP stream then subsequently biased their judgment of the test faces, producing a facial expression aftereffect. It thus raises the question: How did this emotional information extraction occur during the RSVP stream adaptation? 

Haberman and colleagues (2009) reported that participants perceived the sequential presentation of faces by averaging the faces in an RSVP stream through ensemble statistics. This suggests that averaging may be occurring during adaptation to the RSVP stream in the current study. This leads to the question as to whether adaptation to the faces in the RSVP stream would generate an equivalent facial expression aftereffect to an average face created from all of the RSVP faces. We decided to test this hypothesis in Experiment 2. 

Experiment 2

To test whether ensemble perception occurs during adaptation to an RSVP face stream, we adapted the subjects to the face average of the RSVP face stream, and examined whether it generated similar facial expression aftereffects as adapting to the RSVP face stream. If they were to generate facial expression aftereffects of comparable magnitudes, then it would suggest that ensemble statistics may be extracted during the adaptation stage of an RSVP face stream. If it did not occur (for example adaptation aftereffects were smaller to the RSVP stream), then it would suggest that other processes (such as interruption) might occur during adaptation, whereby earlier presented faces in the stream have their perception impaired due to the faces presented later in the stream. 

Methods

Observers

Ten subjects (four males, six females; mean age 22.1 years), with normal or corrected-to-normal vision, participated in Experiment 2. Apart from the subject (HY) who was the experimenter, the other nine subjects were naïve to the purpose of the experiment and different from the subjects in Experiment 1. All subjects gave written consent before testing. This study was approved by the Internal Review Board (IRB) at Nanyang Technological University, Singapore, in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving human subjects. 

Stimuli

All the visual stimuli were the same as those in Experiment 1, except two new face stimuli for adaptation were added. Using the Psychomorph software online (DeBruine & Tiddeman, 2015), we created the average happy face by morphing the 10 happy faces used in the RSVP face stream together. Similarly, we created the average sad face from the 10 different sad faces used in the sad RSVP face stream (Figure 4). These faces, with the same size as the testing stimuli, were used as the adaptors in the Experiment. 

Figure 4

 

The averaged faces used as adaptors in Experiment 2. (A) The averaged face based on all ten happy faces (AF01HAS, AF05HAS, AF07HAS, AF17HAS, AF19HAS, AF20HAS, AF22HAS, AF26HAS, AF29HAS, and AF34HAS). (B) The averaged faces based on all ten sad faces (AF03SAS, AF06SAS, AF07SAS, AF11SAS, AF13SAS, AF20SAS, AF26SAS, AF29SAS, AF32SAS, and AF34SAS).

Apparatus, procedure, and data analysis

The apparatus and data analysis were the same as those in Experiment 1. The general procedure and the trial sequence were similar to those of Experiment 1, except for two new conditions: happy static adaption and sad static adaption. The happy static adaptation used the above generated happy average face as the adaptor (Figure 4A); and the sad static adaptation used the sad average face as the adaptor (Figure 4B). Similar to the RSVP adaptation condition, the static face adaptors were displayed for 3.82 s on the screen in each trial. Therefore, there were five conditions in total: happy static adaptation, sad static adaptation, happy RSVP adaptation, sad RVSP adaptation, and baseline (no adaptation). The subjects went through all five conditions in randomized blocks. Within each block, the test faces were randomly selected from the test face set (Figure 1C) for each participant. 

In contrast to Experiment 1 where each test face was repeated 10 times, each test stimulus was repeated 20 times in Experiment 2. Subjects finished the experiments on two different days within three consecutive days. In the first day, the subjects were tested on all five conditions in randomized blocks with 10 repetitions of each test stimuli, with the same blocks tested on the second day, but this time in a counterbalanced order. 

Results

The results from a naive subject judging the facial expression of the test faces under various conditions are shown in Figure 5A. After adapting to the average happy static face, the subject perceived happy expressions less frequently, and the psychometric curve (solid blue, happy Static) shifted to the right, with virtually the same adaptation aftereffect as after adapting to the happy RSVP face stream (dashed blue, happy RSVP). Similarly, after adapting to the sad average static face, the subject perceived happy expressions more frequently, and the psychometric curve (solid red, sad static) shifted to the left, with similar aftereffects found after adapting to the sad RSVP face stream (dashed red, sad RSVP). This is interesting as it provides the first evidence that an average face generates similar facial expression aftereffects as its RSVP face stream. 

Figure 5

 

The effect of RSVP face stream adaptation and its statistical average face adaptation (Experiment 2). (A) Psychometric functions from a naive subject under the following conditions: Baseline, no adaptation baseline (black circle, solid line); happy RSVP, adaptation to the happy RSVP face stream (blue square, dashed line); sad RSVP, adaptation to the sad RSVP face stream (red diamond, dashed line); happy Static, adaptation to the static average happy face (blue cross, solid line); sad Static, adaptation to the static average sad face (red asterisk, solid line). For each condition, the happy response was plotted as a function of the proportion of happiness of the test face. (B) Summary of all ten subjects' data. For each condition, the average PSE relative to the baseline condition and the SEM were plotted. The p value shown for each condition in the Figure was calculated against the baseline condition using a two-tailed, paired t test.

The summary of PSE shifts from all ten subjects is shown in Figure 5B. Compared to the baseline, all the adaptation conditions generated significant facial expression aftereffects: happy RSVP, M = 0.129, SEM = 0.0196; t(9) = 6.586, p < 0.01; sad RSVP, M = −0.115, SEM = 0.0147; t(9) = −7.744, p < 0.01; happy static, M = 0.127, SEM = 0.0215; t(9) = 5.929, p < 0.01; and sad static, M = −0.134, SEM = 0.0173; t(9) = −7.743, p < 0.01. Further comparisons of the RSVP face streams and their paired average static face conditions revealed that there were no significant differences between them: happy RSVP versus happy static adaptation conditions, t(9) = −0.103, p = 0.920); and sad RSVP versus sad static adaptation conditions, t(9) = −1.388, p = 0.199). Moreover, RSVP adaptation generated similar magnitudes of aftereffects in Experiment 1 and 2: happy, t(18) = 0.844, p = 0.409 and sad, t(18) = −0.920, p = 0.370, RSVP adaptation. It thus suggests that the findings from the first experiment are robust and replicable. 

We then investigated the relationships between the aftereffects from the RSVP face streams and their averaged faces (Figure 6.). We found significant positive correlations between the aftereffects of the happy RSVP face stream and its average happy static face (r = 0.67, p = 0.033), and between the aftereffects of the sad RSVP face stream and its average sad static face (r = 0.64, p = 0.046). The similar trend of a correlation between the two emotions suggests the existence of the same mechanism that drives the static and RSVP face stream adaptation. 

Figure 6

 

The aftereffect of adaptation by RSVP faces as a function of the aftereffect of adaptation by static faces. Ten blue dots represented (upper right corner) 10 subjects' performances on two happy adaptation conditions, and 10 red dots (lower left corner) represented 10 subjects' performances on two sad adaptation conditions. For each emotion, the subjects' adaptation aftereffects to the static and RSVP conditions were significantly correlated.

Discussion

We found that adaption to the RSVP face stream generated similar facial expression aftereffects as adapting to a single average face of the face stream. The magnitudes of their aftereffects were also significantly correlated. This finding supports our hypothesis and indicates that the subjects were temporally averaging the emotions in the face stream through ensemble statistics during adaptation. Studies on spatial visual integration, presenting a set of faces simultaneously, have demonstrated that the visual system interprets this set of faces by involuntarily averaging the spatially distributed faces together (Haberman et al., 2015). Our current findings and previous studies support this view, that temporal averaging occurs during the perception of sequentially presented faces (Haberman & Whitney, 2012). However, we provide new evidence through visual adaptation that if we are exposed to a heterogenous sequence of information (different identities), we are able to average across the different information (i.e., identity), extract the common information (e.g., happy emotion), and that the extracted emotion will subsequently bias our judgement in the relevant tasks (e.g., emotion judgment). This averaging during adaptation is involuntary and implicit, as we did not instruct the subjects to integrate and extract such information during adaptation to the RSVP stream. 

However, all of the face images in our RSVP stream were of the same emotion (happy or sad), without much variation in the strength of the emotion contained within the stream. Average and variability are the two major statistics in a distribution, or sufficient statistics for a normal distribution. Therefore, it raises this following question: Would an adaptation aftereffect still occur with emotional variability in the adapting face stream? To answer this question, we manipulated the emotions in the face sequence in the next two experiments. Experiment 3 introduced variability to the adapting face sequence; and Experiment 4 introduced a single face of unique identity with the opposite emotion (sad) as an outlier, into the RSVP face sequence (happy). 

Experiment 3

Variability in an RSVP face stream may come from two different attributes of the stream: the visual stimuli and the temporal presentation. Stimulus variability can be induced by including different emotions of the face stimuli into the RSVP stream. In contrast, temporal variability can be induced by varying the RSVP's temporal frequencies. We decided to examine the influence of both attributes in the present experiment. We manipulated the variability of the emotions of the adapting face sequence by showing various degrees of happiness in the adapting RSVP stream, from 0.6 to 1.0 in proportion of happiness, while maintaining the average emotion in the RSVP at 0.8 happiness across conditions. Thus there are three types of emotional variance during adaptation: varying from 0.6 to 1.0 randomly; 0.6 or 1.0 binomially; and all faces with 0.8 proportion of happiness. If ensemble perception were to occur during adaptation, we would expect the same magnitudes of facial expression aftereffect (FEA) for all conditions, as they have the same average face during adaptation. However, if some other influences other than ensemble coding were to occur during the RSVP adaptation, we would observe different FEAs for the variability among different conditions. 

Moreover, variation in an RSVP face stream may occur temporally. Increasing temporal frequency may increase temporal variability and reduces the ability to detect a target from an RSVP stream (Potter, 1975, 1976; Potter, Wyble, Hagmann, & McCourt, 2014). However, we do not know whether this will affect adaptation. To this end, we also presented RSVP streams with high (42.5 Hz) or low (5.3 Hz) temporal frequencies. Therefore, if ensemble statistics are still extracted, we would expect no differences between the FEAs for different temporal frequencies. Taken together, strong adaptation despite temporal or emotional variability would provide convincing evidence of ensemble coding. 

Methods

Observers

Ten subjects (three males, seven females; mean age 22.4 years), with normal or corrected-to-normal vision, participated in the experiment. One of the subjects (HY) was the experimenter; the other subjects were naïve to the purpose of the experiment and different from the subjects in previous two experiments. All subjects gave written consent before testing. This study was approved by the Internal Review Board (IRB) at Nanyang Technological University, Singapore, by the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving human subjects. 

Stimuli

All the test stimuli and the raw adapting stimuli were the same as in Experiments 1 and 2. For the additional adapting faces, we morphed between the happy and neutral faces of each of the ten identities (face identities AF01, AF05, AF07, AF17, AF19, AF20, AF22, AF26, AF29, AF34) and generated 10 sets of faces, via the Psychomorph software online (DeBruine & Tiddeman, 2015). In each face set, there were 11 faces, with various proportions of happiness, specifically at 60%, 64%, 68%, 72%, 76%, 80%, 84%, 88%, 92%, 98%, or 100% (Figure 7A). Therefore, there were 110 faces (10 sets × 11 faces) in total. These faces were randomly selected in the RSVP face sequence based on the experiment condition. In addition, we morphed all of these 110 faces into one average face with 80% happiness. We then used this 80% happiness face as the adaptor in the static adaptation condition. 

Figure 7

 

The adaptors used in Experiment 3. (A) Examples of emotional faces used as adaptors. These faces were created by averaging the neutral and the happy faces of one facial identity (AF26). (B) Schematic demonstration for the adapting faces selected on a hypothetical face space. The magenta bar indicates the faces chosen for 80% condition, the blue bars indicate the 60% or 100% condition, and the green square indicates the 60% to 100% condition.

Apparatus, procedure, and data analysis

The apparatus, general trial structure, and data analysis were the same as those in Experiment 2, except there were six new RSVP adaptation conditions, in addition to the static average face adaptation (of the RSVP face sequence) and baseline condition. These six RSVP adaptation conditions varied in two dimensions: (a) emotion variance, and (b) temporal frequency. There were two temporal frequencies: high temporal frequency, with each face displayed on the screen for 23.5 ms (42.5 Hz); and low temporal frequency, with each face displayed for 188 ms (5.3 Hz). There were three types of emotional variances (see Figure 7B): (a) 60% to 100% condition, in which the adapting RSVP stream contains all of the adapting faces except the 80% face, presented in random order; (b) 60% or 100% condition, in which the adapting face stream contains 20 individual faces showing either 60% or 100% happiness; and (c) 80% condition, in which the adapting face stream contains 10 individual faces each showing 80% happiness. Therefore, there were six adaptation conditions whereby emotion and temporal frequency were varied, plus a static average face adaptation condition, and a baseline condition with no adaptation. 

In each condition, each test face was randomly repeated for 20 times. To minimize the fatigue effect, we halved every condition into two identical blocks (similar to Experiment 2). Subjects finished the two separate blocks on two different days within a week. Within each block, the orders of conditions were randomly selected. 

Results

The results from a naïve subject HN judging the facial expressions of test stimuli under all eight conditions are shown in Figure 8A. After adapting to the RSVP stream and the static happy faces (all seven colored lines), the subject was less likely to judge the test face as happy, which were quantified by the rightward shifts of the psychometric curves from the baseline condition (solid black line). 

Figure 8

 

The effects of emotion variation and temporal frequency in face stream (Experiment 3). (A) The psychometric functions from a naïve subject HN under all of the conditions. “Baseline” represents baseline condition without adaptation (black circle, solid line). The “80 Static Average” represents the static adaptation condition with the averaged faces (magenta asterisk, dotted line). The other six lines represent the RSVP conditions. The solid lines represent the high temporal frequency condition, whereas the dashed lines represent the low temporal frequency condition. Magenta, blue, and green lines represent RSVP streams consisting of faces showing 80% happiness, either 60% or 100% happiness, and randomly from 60% to 100% happiness, respectively. For example, the solid magenta line (“80 High RSVP”) indicates the condition with face streams at high temporal frequency, comprising with 80% of happiness. For each condition, the happy response was plotted as a function of the proportion of happiness of the test face. (B) Data summary of all 10 subjects. The white bar (magenta line) indicates the 80 static averaged condition. The full-color bars indicate high temporal frequency conditions (42.5 Hz), and the hatched bars indicate the low temporal frequency conditions (5.3 Hz). Magenta, blue, and green bars represent RSVP streams consisting faces showing 80% happiness, either 60% or 100% happiness, and randomly from 60% to 100% happiness, respectively. For example, the solid magenta bar with caption “80 High RSVP” represents the 80% condition at high temporal frequency. For each condition, the PSE-shift from baseline and the SEM were plotted. The p value shown for each condition above each bar in the Figure was calculated against the baseline condition using the two-tailed, paired t test.

Adaptation aftereffects were calculated as the PSE shift between each adaptation condition and the baseline. The summary of PSE shifts from all 10 subjects is shown in Figure 8B. To investigate the effects of emotional variance and temporal frequency on RSVP adaptation, we performed a two-way, 2 (temporal frequency) × 3 (emotional variance) ANOVA on the facial expression aftereffects (PSE shifts from the baseline) on all six RSVP conditions. We did not find any significant main effect of temporal frequency, F(1, 9) = 1.215, p = 0.299, η_p² = 0.119, or emotional variance, F(2, 18) = 0.517, p = 0.605, η_p² = 0.054. We found no interaction between the two factors either, F(2, 18) = 0.622, p = 0.548, η_p² = 0.065. This suggests that neither emotional variance nor temporal frequency of the RSVP sequence plays an important role in RSVP sequence adaptation. 

To see how these six adaptation conditions compared to the static average face adaptation conditions, we performed a one-way ANOVA which revealed that there was no significant difference among any of the seven adaptation conditions: Greenhouse-Geisser corrected F(3.753, 33.773) = 0.569, p = 0.676, η_p² = 0.059. Moreover, all seven adaptation conditions showed significant adaptation aftereffects (all M > 0.0838, all t > 3.710, all p < 0.005). Further comparisons between the static adaptation condition with each of the RSVP conditions revealed no significant differences (all t < 0.889, all p > 0.397). Adapting to the RSVPs of faces with variable emotions or temporal frequency therefore generated similar facial expression aftereffects as adapting to the static averaged face of the RSVP sequence. 

Discussion

We found that varying the strength of the emotions or temporal frequencies of the RSVP sequence, while maintaining the same average facial emotion, did not lead to any differences in the facial expression aftereffect (FEAs). They all generated the same adaptation aftereffects as their static face average counterpart. Therefore, regardless of the variability in emotion and temporal frequency in the RSVP sequence, as long as the face sequence has the same emotional average, it will still generate the same magnitude of FEAs. 

Unlike Experiment 2 in which all of the adapting faces conveyed emotions at their maximum intensity, here we presented faces with different proportions of happiness. In the 60% or 100% and 60% to 100% conditions, the adaptation aftereffects were not reduced by the presence of the faces with 60% happiness in the adapting face sequence nor increased by the presence of 100% happiness faces in the face sequence. Instead, they both generated similar facial expression aftereffects as the mean emotion (80%) of the face sequence. Moreover, in these two conditions, the mean emotion was not explicitly accessible to the participants; thus any adaptation aftereffects could only be attributed to ensemble statistics. In a separate study, we found that adapting to a dynamic face sequence varying from sad to happy (with the mean emotion as neutral) did not generate any significant aftereffects (Lew & Xu, 2014). Taken together, our findings suggest that the strength of the aftereffects to the RSVP streams was not determined by the variability of the face stream, but was determined by the mean emotion of the stream instead. 

Similar to emotional variance, manipulating the temporal frequency of the RSVP face sequence did not change its facial expression aftereffect: Twenty faces or 160 faces in the face sequence generated the same aftereffect magnitudes. This suggests that extracting the “gist” emotional information from 20 or 160 faces in its average form is the most important factor in the adaptation aftereffect. Therefore, the adaptation aftereffects found here may be explained by temporal ensemble coding (Haberman et al., 2009): Subjects continuously and involuntarily averaged the emotions of the face streams over time and formed a representation of their mean emotion. Whereas we have shown that averaging of emotion occurs across an RSVP stream, it will be of interest to examine whether this still occurs when we match the identity of one of the RSVP faces to the test faces by adaptation. For example, if the matched identity adapting face were to convey a different emotion from the other RSVP faces, then would subjects extract the average emotion of the sequence, or focus on the emotion of the particular identity that matches with the test face? 

Experiment 4

All of the above experiments tested the emotion of an average face identity. It may be the case, however, that the averaging of emotion across an RSVP stream may not be an entirely involuntary process. For example, when the identity of one of the RSVP stream's faces is matched to the identity of the test faces, this face may then override the RSVP stream's ensemble statistics as shown by differential adaptation aftereffects. If individual faces in the face stream play an important role in adaptation, then we would expect to see adaptation aftereffects influenced by a matched identity. On the other hand, if ensemble coding is involuntary and immune to the effects of identity, then the average emotion of the face stream should instead produce matching adaptation aftereffects. 

To examine the influence of facial identity on ensemble encoding, we matched one of the RSVP face's identities to that of the test faces and had it convey the opposite emotion of the other faces in the face stream. For example, if the face stream contained nine happy faces and one sad face, then the sad face adaptor would have the same identity as the test faces. If adaptation is affected by facial identity, rather than averaging, then the aftereffect will be produced by adapting to the face that is matched for identity between the adapting and test faces. Alternatively, if ensemble averaging is immune to the effects of identity, then the adaptation aftereffects to the RSVP stream should be the average emotion of the stream. This result would in turn suggest that ensemble coding of emotion occurs independent of identity. 

Methods

Subjects

Ten new subjects (two males, eight females; mean age 21.9 years), with normal or corrected-to-normal vision, participated in the experiment. One of the subjects was the experimenter (HY). The other subjects were naïve to the purpose of the experiment and different from the subjects in previous experiments. All subjects gave written consent before testing. This study was approved by the Internal Review Board (IRB) at Nanyang Technological University, Singapore, by the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving human subjects. 

Stimuli, apparatus, procedure, and data analysis

The happy and sad face stimuli from Experiment 2 were used in this experiment. We generated three new RSVP streams as the new adapting stimuli: 90% happy, 50% happy, and 10% happy in the face stream. In the 90% happy condition, nine happy faces and one sad face were selected (see Figure 9A highlighted in red). Each face was randomly presented 16 times in the stream. In the 50% happy condition, we randomly selected five happy faces and five sad faces to create the stream. In the 10% happy condition, one happy face and nine sad faces were used. Notably, the happy and sad faces of face identity AF07 were used in all three kinds of RSVP streams. The identity AF07 always showed the opposite emotion to the majority of the stream in the 90% and 10% conditions; and it appeared as happy or sad randomly for half of the time in the 50% happy condition. To minimize low-level adaptation, the adapting stimuli were displayed at 3.20° × 4.03° in size, 133% of the size of test stimuli (Burton et al., 2015; Zhao & Chubb, 2001). 

Figure 9

 

Stimuli used in Experiment 4. (A) The adapting stimuli used for 90% happy condition. This stream contained 10 faces (each repeated 16 times in random orders): nine happy faces (AF01HAS, AF05HAS, AF17HAS, AF19HAS, AF20HAS, AF22HAS, AF26HAS, AF29HAS, and AF34HAS), and one sad face (in the red square, AF07SAS). The sad face was from the same identity (AF07) as the test stimuli. (B) Testing stimuli. These were all generated from one chosen identity (AF07).

The test stimuli were generated from one specific identity (AF07) which was used in the previous RSVP streams. We morphed between the happy and neutral emotional faces of this identity and generated ten sets of faces using Psychomorph software online (DeBruine & Tiddeman, 2015). The proportions of happiness of the test stimuli were at 0%, 10%, 20%, 30%, 40%, 50%, and 60% (Figure 9B). The size of the test stimuli was 2.40° × 3.02°, the same as those in previous studies. 

The apparatus, general trial structure, and data analysis were the same as those in Experiment 3. There were four conditions in total, three RSVP adaptation conditions and baseline. Each test stimuli were tested 20 times randomly in two separate blocks. Therefore, the four conditions were tested in eight blocks with a ten-min rest in between. All participants finished the whole experiment in one day. 

Results

The results from a naïve subject are shown in Figure 10A. The dashed blue line represents adaptation to the 90% happy face stream and shifts the psychometric curve to the right of the baseline condition (solid black line). This result indicates less happy judgments after adapting to the 90% happy face streams, despite the fact that the testing faces were of the same identity as the 10% sad face in the face stream. The dotted red line represents adaptation to the 10% happy face stream (with 90% sad faces in the face stream), and shifts the psychometric curve to the left of the baseline condition. It indicates more happy judgments after adapting to the 10% happy face stream. The dash-dotted magenta line represents adaption to the 50% happy face stream; the psychometric curve did not shift much from the baseline condition. It thus indicates little judgment bias in adapting to 50% happy and 50% sad face stream. 

Figure 10

 

The effects of RSVP face streams with different proportions of happy faces (Experiment 4). (A) The psychometric functions from a naïve subject HD under all four conditions. “Baseline” represents baseline condition without adaptation (black circle, solid line); “90%” represents adapting to RSVP streams with 90% happy full faces (blue cross, dashed line); “50%” represents adapting to RSVP streams with 50% happy full faces (magenta asterisk, dash-dotted line); “10%” represents adapting to RSVP streams with 10% happy full faces (red cross, dotted line). For each condition, the happy response was plotted as a function of the proportion of happiness of the test face. (B) Summary of all 10 subjects' responds. For each condition, the average PSE relative to the baseline condition and the SEM were plotted. The p value shown for each condition in the Figure was calculated against the baseline condition using the two-tailed, paired t test. Blue, magenta, and red bars represent the mean adaptation aftereffects under 90%, 50%, and 10% of happy faces conditions correspondingly.

The summary of results from all 10 subjects is shown in Figure 10B. A one-way ANOVA found that different RSVP streams generated different adaptation aftereffects, F(2, 18) = 21.083, p < 0.001, η_p² = 0.701. Specifically, adapting to the 90% happy face stream generated significant aftereffects, M = 0.0559, t(9) = 4.831, p = 0.001, with adapting to the 10% happy face stream also generating significant aftereffects, M = −0.0754, t(9) = −3.003, p = 0.015. The aftereffects of the two were in opposite directions. Adapting to the RSVP stream with 50% happy faces and 50% sad faces did not generate significant aftereffects, M = −0.0192, t(9) = −1.878, p = 0.093. This suggests that adaptation aftereffects are determined by the mean emotions of the RSVP streams of emotional faces, instead of the individual faces that are of the same identity as the test faces. Therefore, ensemble coding of the facial expressions is independent of identity in the RSVP face stream. 

Discussion

The present experiment found that adapting to a face stream with a majority (e.g., 90%) of happy faces still produced significant facial expression aftereffects in a similar way as adapting to happy faces. Matching the identity of the test faces to the minority (e.g., 10%) sad faces in the face stream did not affect the aftereffects in the same way as adapting to the sad faces. This is because the average face is defined by the overall or majority emotion of the face stream. The minority emotion would therefore seem to be averaged out by the majority emotion. Following this reasoning, adapting to the 50% happy and 50% sad face stream did not produce any significant facial expression aftereffects, as the average emotion of the face stream is neutral. These results offer further evidence to support our hypothesis that ensemble coding of emotion occurs during adaptation to an RSVP face stream. 

When we matched the identity of the test stimuli to one of the identities in the RSVP stream, but presented it with the opposite emotion to the majority of the other faces in that stream, the adaptation aftereffect complied with the majority emotion instead of that of the same identity. During the RSVP face stream, the subjects were able to extract the mean emotion of the face stream, instead of focusing on the emotion of a particular identity which was the same identity as the test faces. It thus suggests that an ensemble representation of the gist emotion is extracted independently of identity when passively viewing a stream of emotional faces. 

General discussion

We conducted four experiments to investigate the ensemble perception of emotional faces in RSVP streams as indexed by visual adaptation aftereffects. Experiment 1 showed that adapting to a set of faces with different identities with the same emotion (happy or sad) presented rapidly affected judgments of emotion in subsequently presented faces. We then generated average faces from the adapting face sets. Experiment 2 demonstrated that the adaptation aftereffects generated by the RSVP face streams are equivalent to, and correlated with, the aftereffects generated by their average static face adaptation counterparts. When varying the emotion and temporal frequency of the face streams in Experiment 3, we found that the RSVP streams induced the same magnitudes of aftereffects as their static averaged counterparts, regardless of the variations in temporal frequency or strength of emotion. Experiment 4 further revealed that ensemble coding of emotion occurred independent of identity in the RSVP stream. When presenting both happy and sad faces of different identities in the RSVP stream, the adaptation aftereffect is dependent on the proportion of the emotion instead of identity in the stream: When half of the faces are sad and half are happy, there are no adaptation aftereffects; when 90% are happy, and 10% are sad, aftereffects are generated in the same direction as adapting to a happy face—and vice versa. Together, these results suggest that we are able to passively average the RSVP faces' emotions during adaptation. It further indicates that we are involuntarily processing this information even when we are not instructed to do so during adaptation. This involuntary processing may be temporal ensemble coding of the face stimuli in RSVP to recalibrate our facial expression norm. 

RSVP and ensemble coding

RSVP paradigms have been used to probe the temporal limit of extracting individual information from a sequence of stimuli (e.g., Keysers, Xiao, Földiák, & Perrett, 2001; Potter, 1976; Potter et al., 2014). The task in an RSVP paradigm is usually to identify or discriminate a target within the sequence, thus showing an effect of masking. The interruption or competition theory may be more appropriate in explaining such findings in these visual masking tasks. In our experiments, although we presented sequences of faces in the RSVP paradigm, we did not ask subjects to remember or recall any specific face. During adaptation, subjects were simply asked to concentrate on the central fixation cross, but not the RSVP face stream in the peripheral region. It has been shown that presenting stimuli in the visual periphery generates larger adaptation aftereffects than when presented at the fovea (on tilt aftereffect, see Chen, Chen, Gao, Yang, & Yan, 2015; on color adaptation, see Bachy & Zaidi, 2014). Therefore, subjects may involuntarily or unconsciously group all of the faces together into an ensemble representation. Coincidentally, because the RSVP sequence is presented during adaptation, ensemble coding of the faces in the sequence may also be able to explain the findings here. Therefore, consistent with previous studies (Haberman et al., 2009; Haberman et al., 2015; Haberman & Whitney, 2009), we found that subjects integrated the consecutively encountered faces together into their face average during adaptation. 

Our results in all experiments suggest that ensemble representations are extracted involuntarily during adaptation. When the subjects were adapted to the RSVP of faces, the subsequent aftereffect was influenced by the faces presented in the RSVP stream; the similar aftereffects from adapting to the RSVP face sequence and its static average face suggests that the subjects involuntarily integrate all of the faces together to create an average face. Moreover, strong correlations between the aftereffects of RSVP of faces and the static faces observed in Experiment 2, as well as the similar adaptation aftereffects between RSVP streams and the static averaged faces in Experiment 3, show that there are similar mechanisms between the processing of the RSVP of faces and the static faces, i.e., the magnitude of the RSVP face stream adaptation aftereffect could be predicted from the magnitude of the static face adaptation (Haberman et al., 2015), thus further supporting the notion that the visual system interprets the RSVP of faces as the statistically averaged static face (Haberman et al., 2009). 

Our experiments in temporal ensemble representations resemble other studies in spatial ensemble representations of faces. It has been shown that spatial ensemble coding occurs when seeing a group of faces on the screen (Fischer & Whitney, 2011; Haberman et al., 2015; Haberman & Whitney, 2007, 2009, 2010). Moreover, ensemble coding makes it effortless to glean the gist of all faces' emotional information, despite being unable to explicitly identify individual faces separately (e.g., Haberman & Whitney, 2007). Therefore, both temporal and spatial ensemble coding of faces might form an ensemble representation of the whole group of faces at the cost of discriminating individual faces in the group. 

Identity versus facial expression processing

Our experiments showed that adapting to a stream of faces of different identities generated a strong and significant facial expression aftereffect, and that this aftereffect was determined by the emotion of the average face of the face stream. Previous studies have shown that after adapting to one identity posing a certain emotion, participants will then produce robust emotion adaptation aftereffects when judging emotion in subsequently presented faces of different identities; in other words, there are identity-independent emotion processing mechanisms (Fox & Barton, 2007; Campbell, & Burke, 2009). Note that the test faces in the first three experiments were of the average face from the adapting face stream; therefore, the adapting and test faces were of different identities. Moreover, the manipulation in Experiment 4 confirmed that it is the emotion, instead of the identity of the face stream, which affects facial expression aftereffects. Further evidence from developmental prosopagnosia cases, who have lifelong difficulties in facial identity recognition, show similar emotion recognition abilities as neurotypical participants (Duchaine, Parker, & Nakayama, 2003; Humphreys, Avidan, & Behrmann, 2007; Palermo et al., 2011; but also see Biotti & Cook, 2016). This independence of facial identity and emotion is supported by models of face perception that posit the existence of distinct cortical regions involved in the separate processing of facial identity and emotion (Haxby, Hoffman, & Gobbini, 2000, 2002; Haxby & Gobbini, 2011). Therefore, our current findings in adaptation aftereffects of RSVP are in line with these previous studies indicating identity-independent facial emotion processing. Future neuroimaging and TMS studies will be required, however, to identify the cortical regions functionally involved in this averaging of facial emotion during RSVP. 

Functional accounts of adaptation and ensemble coding

According to Skinner and Benton (2010), Webster and MacLeod (2011), and Xu et al., (2008), psychologically, visual adaption leads to perceptual bias (the subsequent face's characteristics are perceived in the opposite direction to the adapted face's attributes), increased sensitivity (the sensitivity to the stimulus near the adaptor increased), and normalization (the adapting stimulus appears less extreme in its attribute). The functional accounts of adaptation range from efficient coding and information optimization (increase in sensitivity after adaptation, Clifford et al., 2007; Rieke & Rudd, 2009; Stocker & Simoncelli, 2006; Wainwright, 1999; Wark, Lundstrom, & Fairhall, 2007; and decorrelation of neural activities to different stimulus attributes, Barlow, 1990; Mather, Verstraten, & Anstis, 1998), error correction (perception of the stimuli is adjusted to reflect the change in the visual stimuli, Andrews, 1967), to recalibration (encode the nature of the world through sensory messages; normalization of the neural response for the mean stimulus level, Webster, 2014; and build predictions about the world, Chopin & Mamassian, 2012; Srinivasan, Laughlin, & Dubs, 1982). Our current study observed perceptual bias, such that adapting to a happy face sequence biased the subsequently presented faces towards less happy. In addition, this bias was determined by the statistical average face of the adapting RSVP face stream, through a recalibration or normalization process. Together, these findings show that adaptation to emotional faces presented in an RSVP stream over time is similar to the adaptation to a single average face, suggesting extraction of average face, or ensemble coding, is involuntary. 

On the other hand, recent studies reported that four-year-old children are able to form ensemble representations (Sweeny, Wurnitsch, Gopnik, & Whitney, 2015), and can use norm-based coding for faces (Jeffery, Read, & Rhodes, 2013). Our study, together with these studies (see also Rhodes, Neumann, Ewing, & Palermo, 2015), highlight a potential link between ensemble statistics and norm-based face processing. Our current findings delineated the formation of the facial expression norm by adaptation: Our vision system implicitly integrates the multiple faces we encounter over time to the average face of the face stream, an ensemble representation updated norm of the facial expression space shaped by recent visual experience. Therefore, our findings, for the first time, offer a new insight into the updating procedure of the facial expression norm and involuntary face processing. 

Conclusions

We have shown here that rapidly presented face sequences of emotional faces are perceived as the averaged face involuntarily. The facial expression aftereffects can be generated by adapting to an RSVP face stream, and these adaptation aftereffects are similar to those generated by adapting to the statistical average face of the RSVP face stream. Furthermore, the representation of the RSVP stream, quantified by the adaptation aftereffect, was determined by the mean emotion of the stream, but not by the temporal frequency or the emotional variance. Therefore, the individual face information may be compromised, but the average emotion of the face sequence is clear and sound, probably through ensemble coding by statistical averaging of the faces in the sequence during adaptation. 

Acknowledgments

This work was supported by Nanyang Technological University Research Scholarship (HY); College of Humanities, Arts, and Social Sciences Incentive Scheme (HX), Singapore Ministry of Education Academic Research Fund (AcRF) Tier 1 (HX). We thank Dr. Edwin Burns for his helpful comments on the manuscript. 

Commercial relationships: none. 

Corresponding author: Hong Xu. 

Email: [e-mail]. 

Address: Division of Psychology, School of Humanities and Social Sciences, Nanyang Technological University, Singapore.