Transfer of the face viewpoint aftereffect from adaptation to different and inverted faces

Abstract

The viewpoint aftereffect is a perceptual illusion that, after adapting to an object/face viewed from one side (e.g., 30° to the left of center), when the same object/face is subsequently presented near the front view, the perceived viewing direction is biased in a direction opposite to that of the adapted viewpoint (e.g., 2° to the right). In this study, we measured the face viewpoint aftereffects when the adapting and the testing faces were different in identity and gender and when their vertical orientations were inverted. The aftereffect showed a strong transfer following adaptation to other faces. This effect was slightly attenuated when the adapting and the test face stimuli were made more dissimilar. This suggests the existence of neurons jointly tuned to both face view and structure. However, the transfer from cross adapting to an inverted face was much reduced and weak, indicating that the neural coding of upright and inverted faces in the high-level visual cortex is different and a major part of the face viewpoint coding occurs at the level where faces are holistically represented.

Introduction

After visual adaptation to an object (e.g., a face) viewed from one side, the perceived viewing direction of the same object/face subsequently presented near its front view is biased in a direction opposite to that of the adapted viewpoint ( Figure 1). This aftereffect was termed viewpoint aftereffect. Observation of such a viewpoint aftereffect supports the existence of viewer-centered object representation in the human visual system (Fang & He, 2005; Ryu & Chaudhuri, 2006). Our previous study has shown that there was a very weak or a little transfer of the viewpoint aftereffect between objects from two different categories (e.g., no cross adaptation between a face and a paperclip), suggesting that viewpoint information is coded specifically for object categories. However, little is known about how sensitive the viewpoint aftereffect is to differences between objects within a category. Thus, the first question we addressed in this study is how the structural similarity between adapting and test faces modulates the viewpoint aftereffect. In the first and second experiments, we generated the adapting and the test stimuli through face morphing along identity dimension (Leopold, O'Toole, Vetter, & Blanz, 2001) and gender dimension (Webster, Kaping, Mizokami, & Duhamel, 2004), respectively. The difference in gender dimension is not just another case of identity difference. Gender forms a natural category and tends to be more salient than almost any other feature when looking at a face. Studying the neural coding of gender information is becoming an important topic in face perception (Ng, Ciaramitaro, Anstis, Boynton, & Fine, 2006). Viewpoint aftereffect transfer depends largely on whether view-selective face neurons in the human visual system are also tuned to face structures (e.g., identity and gender). Perrett, Hietanen, Oram, Benson, and Rolls (1992) observed that most view-selective face neurons in monkeys' inferior temporal cortex are not sensitive to identity, but identity-sensitive neurons are often view selective. From this evidence, we would predict a strong but not complete transfer of the face viewpoint aftereffect when adapting and test faces are of different identities. Other vision researchers investigated this joint neural coding of face structure and view using face configural aftereffect (Leopold et al., 2001; Webster et al., 2004). They tested whether the face configural aftereffect can transfer across a substantial change in viewpoint, but there remains no consensus that face structure is view-specific coded (Anderson & Wilson, 2005; Jeffery, Rhodes, & Busey, 2006; Jiang, Blanz, & O'Toole, 2006). 

Figure 1

 

Schematic description of a face viewpoint adaptation experiment. After a 25-s preadaptation and a 5-s topping-up adaptation to a 30° side view of a face, a test stimulus (front view or close to front view) was presented briefly. Subjects were asked to make a two-alternative forced-choice (2-AFC) judgment of the viewing direction of the test stimulus, either left or right.

The second question we addressed in this study is how the vertical orientation of the adapting and test faces modulates the viewpoint aftereffect. In the third experiment, subjects were asked to adapt to an inverted face, but test with an upright face. This experiment will provide further insight into the neural representation of inverted faces, which is still poorly understood. The face inversion effect (FIE), the phenomenon that is defined as the larger decrease in face recognition performance than for other mono-oriented objects when presented upside-down (Rossion & Gauthier, 2002), suggests that the face perception system is not engaged effectively by inverted faces. However, functional magnetic resonance imaging (fMRI) studies have found little or very weak FIEs in the face-selective areas, including the occipital face area (OFA), fusiform face area (FFA), and superior temporal sulcus (STS; Haxby et al., 1999; Kanwisher, Tong, & Nakayama, 1998; Yovel & Kanwisher, 2005), and an image classification study by Sekuler, Gaspar, Gold, and Bennett (2004) also suggested that inverted face processing is only quantitatively, not qualitatively different from upright face processing. A recent behavioral study (Rhodes et al., 2004) shows that opposite face configural aftereffects can be simultaneously induced for upright and inverted faces (called orientation-contingent face aftereffect), demonstrating that distinct neural populations code upright and inverted faces. Similarly, we investigated this issue using the face viewpoint aftereffect. The transfer of face viewpoint aftereffects between inverted and upright faces would depend on the overlap of the neural coding mechanisms engaged by upright and inverted faces. 

Experiment 1: Transfer of the face viewpoint aftereffect from adaptation to different faces morphed along the identity dimension

The goal of this experiment is to check whether face viewpoint is coded jointly with face identity. The transfer (or lack thereof) of a face viewpoint aftereffect is taken as an indicator of joint coding. Specifically, the face viewpoint aftereffect was measured with a test face of one identity following adaptation to a face with varying degrees of identity difference. 

Method

Apparatus and stimuli

Stimuli were presented on a SONY Trinitron Multiscan G420 19-in. monitor, with a spatial resolution of 1280 × 1024 and a refresh rate of 100 Hz. The viewing distance was 57 cm. The adapting and the test stimuli were generated by projecting a three-dimensional face model with different in-depth rotation angles onto the monitor plane with the front view as the initial position; 30° rotation for adaptors; and 0°, 3°, and 6° rotation for test stimuli. Both left and right rotations were executed. The three-dimensional face models were generated by FaceGen Modeller 3.1 ( http://www.facegen.com/) based on a series of face images morphed along the identity dimension defined by Jim and Anti-Jim, developed by the Max Planck Institute for Biological Cybernetics (Leopold et al., 2001). The adapting stimuli were the 30° side views (left and right) of Jim100, Jim50, and Anti-Jim50 (Figure 2A). The average face is presented here to show a morphing transition. The test stimuli were the front view (0°) and 3° and 6° side views (left and right) of Jim100 (Figure 2B). Numbers following Jim or Anti-Jim refer to the identity strength possessed by the given face. All the stimuli extended no more than 3.2° × 3.2°. 

Figure 2

 

Transfer of the face viewpoint aftereffect from adaptation to different faces morphed along the identity dimension. (A) Adapting stimuli are the 30° side views (left and right) of Jim100, Jim50, and Anti-Jim50. The average face is presented here to show a morphing transition. (B) Test stimuli are the front view (0°) and 3° and 6° side views (left and right) of Jim100. (C) Psychometric functions showing viewing direction judgments without adaptation and after adapting to different faces. Data points were fit using a cumulative normal function. The abscissa refers to the five views of test stimuli. Zero degree is the front view and S6, S3, O3, and O6 are side views ±3° or ±6° away from the front view. S and O indicate that the test stimulus has the same or opposite viewing direction (left or right) as the adaptor, respectively. The ordinate refers to the percentage of trials in which subjects indicated that the viewing direction of the test stimulus was opposite to the adaptor. Error bars denote 1 SEM.

Subjects

Seven naive subjects (three male and four female) with normal or corrected-to-normal vision participated in the first experiment. They gave written, informed consent in accordance with procedures and protocols approved by the human subject review committee of the University of Minnesota. 

Procedure

There were three adaptation conditions and one baseline condition. Each condition had eight blocks and each block consisted of 50 trials. For the three adaptation conditions, subjects adapted to the 30° side view of Jim100, Jim50, and Anti-Jim50, respectively (four blocks with the left side view adaptor and the other four with the right side view adaptor), but the five test stimuli were always the front view and 3° and 6° side views (left and right) of Jim100. Each adaptation block began with a 25-s preadaptation. After a 5-s topping-up adaptation and a 1-s blank interval, one of the five test stimuli was presented for 0.2 s and subjects were asked to make a two-alternative forced-choice (2-AFC) judgment of the viewing direction of the test stimulus, either left or right ( Figure 1). To avoid local adaptation during the adaptation period, the adapting stimulus floated randomly within a 5.7° × 5.7° area, whose center was coincident with the center of the monitor. The starting point of the adapting stimulus was randomly distributed in this 5.7° × 5.7° area, and its floating velocity was 0.85 deg/s. The position of the test stimulus was also randomly distributed within the 5.7° × 5.7° area. During the experimental period, a fixation point was placed in the center of the monitor and subjects were required to maintain fixation. In each adaptation block, each of the five test stimuli was presented 10 times, for a total of 50 stimulus presentations/trials and with a random sequence. All of the data from the eight blocks were pooled together for analysis. The baseline condition was very similar to the adaptation conditions except that subjects were asked to judge the viewing direction of the test stimulus without any adaptation. The temporal order of a total of 32 (8 × 4) blocks was randomized across four experimental conditions. Subjects were given one practice block for each experimental condition before the main experiment. 

Results

The results are presented in Figure 2 as psychometric functions: The percentage of trials in which subjects indicated that the viewing direction of the test face was opposite to the adaptor plotted as a function of test stimulus true viewpoint. Without any adaptation, subjects gave nearly perfect performance for all five test stimuli (50% level for the front view, correct identification for the 3° and the 6° test stimuli; see the black line in Figure 2C). In other words, subjects had no trouble discriminating viewing directions of 3° and 6° from the front view. However, after a 25-s preadaptation and a 5-s topping-up adaptation to the 30° side view of faces, the psychometric function showed a general horizontal shift to the left (compare black and red, green, blue lines in Figure 2C). The front views were often judged as facing away from the adapted viewing direction, and even some of the test stimuli facing in the same direction as the adaptors were perceived as facing the direction opposite that of the adaptors. To quantitatively measure the magnitude of the face viewpoint aftereffect, the psychometric values at the five test views were fit by using a cumulative normal function. We interpolated to find the view expected to be seen as the front view in 50% of the trials before and after adaptation. We quantified the magnitude of the face viewpoint aftereffect as the angular difference between the views found through interpolation before and after adaptation (i.e., horizontal shift between the cumulative normal functions). As shown in Figure 2C, we saw a gradual decrease in the magnitude of the face viewpoint aftereffect when the adapting and the test stimuli became more and more dissimilar (adaptor: mean ± SEM; Jim100: 2.11° ± 0.31°; Jim50: 1.93° ± 0.22°; Anti-Jim50: 1.59° ± 0.17°). We performed a repeated measures ANOVA of the magnitudes to examine the effect of the identity strength of the adaptors (Jim100, Jim50, and Anti-Jim50) on this aftereffect. We found a significant main effect, F(2, 14) = 6.438, p = .013. Post hoc LSD tests further revealed a significant difference between Jim100 and Anti-Jim50, t(6) = 2.728, p = .034, and between Jim50 and Anti-Jim50, t(6) = 2.957, p = .025. 

Experiment 2: Transfer of the face viewpoint aftereffect from adaptation to different faces morphed along the gender dimension

The goal of this experiment is to check whether face viewpoint is coded jointly with face gender. We measured the face viewpoint aftereffect using a test face following adaptation to faces with varying degrees of gender difference. 

Method

Seven naive subjects (three male and four female) with normal or corrected-to-normal vision participated in the second experiment. Four of them also participated in the first experiment. The equipment and the basic procedure were the same as those used in the first experiment; however, the adapting and the test stimuli used in the second experiment were morphed along the gender dimension. The adapting stimuli were the 30° side views (left and right) of Male100, Male50, and Female50 ( Figure 3A). The average face is presented here to show a morphing transition. The test stimuli were the front view (0°) and 3° and 6° side views (left and right) of Female100 ( Figure 3B). Numbers following male or female refer to the gender strength possessed by the given face. 

Figure 3

 

Transfer of the face viewpoint aftereffect from adaptation to different faces morphed along the gender dimension. (A) Adapting stimuli are the 30° side views (left and right) of Male100, Male50, and Female50. The average face is presented here to show a morphing transition. (B) Test stimuli are the front view (0°) and 3° and 6° side views (left and right) of Male100. (C) Psychometric functions showing viewing direction judgments without adaptation and after adapting to different faces. Data points were fit using a cumulative normal function. Error bars denote 1 SEM.

Results

As shown in Figure 3C, it is evident that there were strong face viewpoint aftereffects after adaptation and the magnitude gradually decreased when the adapting and the test stimuli became more and more dissimilar (adaptor: mean ± SEM; Male100: 2.19° ± 0.16°; Male50: 1.84° ± 0.12°; Female50: 1.61° ± 0.1°). To examine the effect of the gender strength (Male100, Male50, and Female50) of the adaptors on this aftereffect, we performed a repeated measures ANOVA of the magnitudes. We found a significant main effect, F(2, 14) = 11.912, p = .001. Post hoc LSD tests further revealed a significant difference between Male100 and Male50, t(6) = 2.728, p = .034, and between Male100 and Female50, t(6) = 2.957, p = .025. 

Experiment 3: Transfer of the face viewpoint aftereffect from adaptation to an inverted face

The goal of this experiment is to check the overlap of the neural coding mechanisms engaged by upright and inverted faces by measuring the transfer of the face viewpoint aftereffect from adaptation to an inverted face. 

Method

The subjects, equipment, and basic procedures were the same as those used in the second experiment, with the exception of the adapting and the test stimuli. The adapting stimuli were the 30° side views (left and right) of an upright face and its vertical inversion ( Figure 4A). The test stimuli were the front view (0°) and 3° and 6° side views (left and right) of the upright face ( Figure 4B). 

Figure 4

 

Transfer of the face viewpoint aftereffect from adaptation to an inverted face. (A) Adapting stimuli are the 30° side views (left and right) of upright and inverted faces. (B) Test stimuli are the front view (0°) and 3° and 6° side views (left and right) of the upright face. (C) Psychometric functions showing viewing direction judgments without adaptation and after adapting to upright and inverted faces. Data points were fit using a cumulative normal function. Error bars denote 1 SEM.

Results

As shown in Figure 4C, we see a strong face viewpoint aftereffect after adaptation to the upright face. The magnitude significantly decreased (but was still substantial) when an inverted face was used as an adaptor (adaptor: mean ± SEM; upright: 1.97° ± 0.25°; inverted: 0.86° ± 0.24°). A repeated measures ANOVA of the magnitudes was performed to examine the effect of the orientation (upright and inverted) of the adaptors on this aftereffect. We found a significant main effect, F(1, 7) = 21.536, p = .004. 

To compare the viewpoint aftereffect transfer across these three experiments, we normalized the magnitude of the face viewpoint aftereffect from cross adaptation for each experiment (for Experiment 1, adapt to Anti-Jim50 and test with Jim100; for Experiment 2, adapt to Female50 and test with Male100; for Experiment 3, adapt to inverted face and test with upright face) by dividing it by the magnitude of the face viewpoint aftereffect when the adapting and test stimuli were of the same identity or orientation. In all three of these experiments, the absolute magnitudes of the face viewpoint aftereffect when the adapting and the test stimuli were of the same identity or orientation were very similar. However, the decrease in magnitude was greater from cross adapting to the inverted face. With one-way ANOVAs performed on the normalized face viewpoint aftereffects, we found that the aftereffect from cross adapting to the inverted face was much weaker than those from cross adapting to Anti-Jim50, F(1, 28) = 6.916, p = .022, and Female50, F(1, 28) = 7.245, p = .02, but there was no significant difference between Anti-Jim50 and Female50, F(1, 28) = 0.499, p = .493 ( Figure 5). These comparisons demonstrated that the face viewpoint aftereffect was more sensitive to the adaptor's orientation change than its identity and gender change, but that no qualitative difference existed between the latter two conditions. 

Figure 5

 

Normalized face viewpoint aftereffect from cross adaptation. When the adapting and the test stimuli were generated from the same face model (Jim100 or Male100) or the same orientation (upright), the magnitude of the viewpoint aftereffect was set to 1. Error bars denote 1 SEM.

Discussion

We found a strong transfer of the face viewpoint aftereffect from adaptation to different faces along both the identity and the gender dimensions. This transfer attenuated when the adapting and test stimuli became more and more dissimilar. We also found that adapting to an inverted face resulted in a weak transfer of the face viewpoint aftereffect. It is unlikely that these results can be explained by low-level retinotopic adaptations. First, the positions of the adapting and the test stimuli were randomized. Second, the adaptor was randomly and slowly drifting during preadaptation and topping-up adaptation. Third, the inverted face has the same local features as the upright face, but its adaptation effect was significantly weaker than the other faces' adaptation effects. Therefore, we believe that these results reflect high-level neural adaptation and have important implications about the representations of upright and inverted faces in the human visual system. 

The strong but incomplete transfer of the face viewpoint aftereffect from adaptation to different faces suggests that some face view-selective neurons in the human visual system are also tuned to face structures (e.g., identity and gender). If this were not the case, we should have observed a complete transfer. These results are in line with view-specific coding of face shape (Jeffery et al., 2006). Recent fMRI adaptation studies have demonstrated that both the FFA and the STS in the human visual system contain face view-selective neurons (Andrews & Ewbank, 2004; Fang, Murray, & He, 2007). However, only the FFA shows face identity adaptation (Andrews & Ewbank, 2004) and the FFA is also believed to support face identification (Grill-Spector, Knouf, & Kanwisher, 2004). Converging evidence suggests that the STS might be more responsible for the strong transfer of the face viewpoint aftereffect and the FFA might be more responsible for the slight attenuation of this aftereffect due to cross adaptation. 

An alternative explanation of the transfer attenuation from adaptation to different faces along the identity and the gender dimensions is that the magnitude of the face viewpoint aftereffect depends on the image similarity between the adapting and test stimuli. We cannot completely disentangle the image similarity from the morphing manipulation along the identity and the gender dimensions because we have to gradually change the images to generate the gender and the identity differences. That is, the image similarity is the independent variable we manipulated. However, in our previous study (Fang & He, 2005), we have demonstrated that the viewpoint aftereffect is a global three-dimensional effect, rather than a two-dimensional pattern/feature adaptation effect. In these experiments, although the two adapting stimuli were very similar or almost identical in their image features and two-dimensional patterns, the magnitudes of the viewpoint aftereffects were dramatically different, depending on their global three-dimensional explanations. The weak aftereffect from adaptation to the inverted face also contradicts the image similarity explanation because the inverted and the upright faces have the same local features. Recent fMRI studies (Andrews & Ewbank, 2004; Fang et al., 2007) found that face viewpoint adaptation occurred in the FFA and the STS, not in the LOC, a shape-selective area. This evidence suggests that the observed adaptation effect in this study most likely occurred at a holistic processing stage, not at a form/shape processing stage (Webster & MacLin, 1999). 

The weak transfer of the face viewpoint aftereffect from inverted adapting faces is consistent with the orientation-contingent face aftereffect (Rhodes et al., 2004) and a finding that the face configural aftereffect significantly decreased when the adapting stimulus was an inverted face and the test stimulus was an upright face (Watson & Clifford, 2003; Webster & MacLin, 1999). This result suggests that the neural coding of upright faces and inverted faces in the high-level visual cortex is quite different, and a major part of face viewpoint coding occurs at a level where faces are holistically represented. Haxby et al. (1999) found house-selective areas in the human inferior temporal cortex engaged in the representation of inverted faces, which suggests that inverted faces might be processed like general objects. However, the face-selective areas, OFA, FFA, and STS (Haxby et al., 1999; Kanwisher et al., 1998; Yovel & Kanwisher, 2005) showed a little or very weak FIE in fMRI studies. This might be due to the fact that fMRI signals in the visual cortex can be easily modulated by high-level cognitive processes (e.g., mental imagery). For example, Cox, Meyers, and Sinha (2004) showed that the FFA maintains its selectivity for faces without regard to whether the faces are defined intrinsically or contextually. After all, inverted faces can be readily identified as faces and finally recognized at an individual level. These processes cannot be easily revealed by fMRI given its poor temporal resolution. However, event-related potential studies in humans (Bentin, Allison, Puce, Perez, & McCarthy, 1996) and single-unit recordings in monkeys (Perrett et al., 1985) have found that latencies of neural responses to inverted faces are delayed compared to upright faces. With fMRI techniques, a solution to this paradox is fMRI adaptation (Fang et al., 2007). This psychophysical adaptation paradigm can be exactly adopted to investigate the FIE in human visual cortex. Yovel and Kanwisher (2005) used a short-term (250 ms, rather than 30 s in this study) adaptation method to demonstrate that the FFA is a primary neural source of the behavioral FIE. 

The current study together with our previous study (Fang & He, 2005) demonstrates a strong transfer of the viewpoint aftereffect between different faces; a weak, but substantial, transfer between upright and inverted faces; and little transfer between exemplars from different object categories (e.g., face and paperclip). It seems that the transfer of the viewpoint aftereffect between two objects is proportional to the overlap of their neural representation. Adaptation is often called the psychophysicist's “electrode.” This cross-adaptation paradigm might be able to provide an objective way to measure “neural distance” between different objects, and even to categorize objects. It can be a powerful tool for exploring the computational mechanisms of face and object coding. However, two caveats should be noted. First, we should be cautious about making a comparison between cross-adaptation effects along different stimulus dimensions (e.g., face orientation change vs. identity change) and drawing a strong conclusion from it. Second, the “neural distance” idea is still unfledged. We need more psychophysical and fMRI data based on different types of stimuli to test it in the future (e.g., demonstrate a correlation between lack of aftereffect transfer and a release of fMRI adaptation). 

Acknowledgments

We thank David Leopold for providing the face images with different identities and Serena Thompson and Robert Shannon for their helpful comments. This research was supported by the James S. McDonnell Foundation and NIH R01 EY015261-01. 

Commercial relationships: none. 

Corresponding author: Fang Fang. 

Email: [e-mail]. 

Address: Department of Psychology, Peking University, 5 Yiheyuan Road, Beijing 100871, PR China.