Neurophysiological (W. A. Freiwald, D. Y. Tsao, & M. S. Livingstone, 2009; D. A. Leopold, I. V. Bondar, & M. A. Giese, 2006) and psychophysical (D. A. Leopold, A. J. O'Toole, T. Vetter, & V. Blanz, 2001; G. Rhodes & L. Jeffery, 2006; R. Robbins, E. McKone, & M. Edwards, 2007) evidence suggests that faces are encoded as differences from a mean or prototypical face, consistent with the conceptual framework of a mean-centered face space (T. Valentine, 1991). However, it remains unclear how we encode facial similarity across classes such as gender, age, or race. We synthesized Caucasian male and female cross-gender “siblings” and “anti-siblings” by projecting vectors representing deviations of faces from one gender mean into another gender. Subjects perceived male and female pairings with similar vector deviations from their gender means as more similar, and those with opposite vector deviations as less similar, than randomly selected cross-gender pairings. Agreement in relative direction in a space describing how facial images differ from a mean can therefore provide a basis for perceived facial similarity. We further demonstrate that relative coding for male and female faces is based on the activation of a shared neural population by the transfer of an identity aftereffect between a face and its cross-gender sibling. These results imply whereas structural similarity may be reflected in the Euclidean distance between points in face space configural similarity may be coded by direction in face space.

*identity trajectory*of a face is the line that extends through that face from the mean. Faces on this trajectory have specific perceptual relationships to the original face. Faces between the mean (identity strength = 0) and the original face (identity strength = +1) are attenuated (

*anti-caricatured*) versions of the original face. Faces beyond the original face (identity strength > +1) are exaggerations (

*caricatures*) of the original face. Projecting the original face through the mean generates an

*anti-face*(identity strength = −1), which is perceptually opposite to the original face.

*F*as an example):

- Subtracting the mean female morphed image vector (
**F**) from the sample face morphed image vector (*F*) to give the female mean relative vector (*F*_{ F }); - Finding the dot product of
*F*_{ F }with the (60) vectors of the male basis set. This gives a set of 60 PC coefficients; - Multiplying the vectors of the male basis set with the PC coefficients to give the male mean relative vector
*F*_{ M }; - Adding the male mean vector (
**M**) to*F*_{ M }to give the vector of the synthesized male “sibling” of*F*designated as*M*; - Reconstructing
*M*from vector to image form.

*F*

_{ F }) is calculated in the morphed image space, not in the reduced dimensionality PCA space. However, the female mean relative vector (

*F*

_{ F }) may not lie in the subspace describing male face variability once shifted to be relative to the male mean (

**M**). To ensure the reconstruction looks male, without image artifacts, we need to project the female mean relative vector into the male PCA space and reconstruct the vector (

*F*

_{ M }). This new male mean referenced vector will not necessarily be parallel to the original female mean relative vector (

*F*

_{ F }) in the morphed image space. Since the bases of the male and female spaces are different and the ordering of the bases has been based, conventionally, on the amount of variance accounted for, the directions in the two spaces cannot be compared although the directions of mean relative vectors (

*F*

_{ F }and

*F*

_{ M }) in the morphed image space can be compared. Ultimately, this disparate ordering of components does not matter for synthesis, as synthesis involves summing all the independent projections of

*F*

_{ F }onto the PCA components of the male basis set to give the PC coefficients of

*F*

_{ M }.

*sibling pair*); the distractor pair comprised an original face and a randomly selected cross-gender synthesized face. In Experiment 1b, the distractor pair comprised an original face and the cross-gender anti-face synthesized from it (

*anti-face pair*); the target pair comprised an original face and a randomly selected cross-gender synthesized face. The position (top/bottom) and gender of the original members of each pair and the side (left/right) of the target and distractor pairs was counterbalanced. Each subject ran 2 blocks of 40 trials of either Experiment 1a or 1b; in one block, the faces were inverted.

*t*-test; performance on upright and inverted faces was compared using paired

*t*-tests.

- Morphed image vector Euclidean distance: the square root of the sum of the squared differences between the equivalent units of the mean relative vectors representing each face in morphed image space.
- Image-based Euclidean distance: the square root of the sum of the squared differences between equivalent pixel intensity values in each face.

*t*-tests, all

*p*< 0.05) except inverted faces in Experiment 1a in which accuracy was marginally above chance (

*p*< 0.06). As shown in Figure 2, subjects chose the target pair more often when the faces were upright than when they were inverted (paired

*t*-tests, 2-tailed,

*df*= 11, Experiment 1a,

*t*= 5.83,

*p*< 0.0001; Experiment 1b,

*t*= 6.42,

*p*< 0.00005). Judgments of similarity might tap a generic pattern matching mechanism; however, the highly significant inversion effects indicate that subjects' judgments were based on facial configurations, not simply on facial features. There were no differences between response latencies in any condition (all

*p*> 0.3; paired

*t*-test) except for inverted faces in Experiment 1a in which target responses were faster than distractor responses (

*p*= 0.01).

*r*(78) = 0.284,

*p*= 0.011, not corrected for multiple comparisons), but no other correlations were significant (all

*p*> 0.1). The correlation between pixelwise image similarity and accuracy for inverted faces in Experiment 1b may have emerged because neither pair appeared similar (because they were an anti-face pairing and a random pairing). In the absence of face-specific mechanisms as used in upright face processing, subjects fell back on basic measures of image similarity. However, it is clear that in upright conditions, in which face-sensitive configural processing mechanisms are dominant, performance was not based on simple feature comparison or image similarity.

*Angela*+ 0 ×

*Barbara*to 0 ×

*Angela*+ 0.7 ×

*Barbara*(see Figure 3). Female target and test faces subtended a visual angle of approximately 10.2° × 8.2°. Male adapting faces subtended a visual angle of approximately 10.8° × 8.6°. Fixation point, training, and adapting faces were presented centrally; test faces were presented at 8 locations on a circle 8.2° from fixation. Faces were presented on a black background rectangle on a mid-gray screen.

*baseline*condition) or to the male anti-face of one of the target female faces (

*adapted*condition) in each block (A–B–B–A design). Each block contained 192 trials: 8 trials per level per target axis (A/B, A/C, B/C) giving 16 trials for each data point in both conditions. Each subject was adapted to only one anti-face (counterbalanced across subjects: 5 to male anti-Angela, 4 to male anti-Barbara, 4 to male anti-Carol).

*baseline*: following exposure to the male mean and

*adapted*: following exposure to a male anti-face). Cumulative Gaussian functions were fitted using psignifit (see http://bootstrap-software.org/psignifit/; Wichmann & Hill, 2001). For each axis, equal upper and lower slip rates were calculated using combined baseline and adapted data. Separate functions were then fitted for each condition using these slip rates and the point of subjective equality (PSE) extracted for each function. We predict that, compared to baseline, adaptation to the anti-face will shift the PSE on axes involving the adapting (anti-)face toward the female identity from which the male adapting anti-face was derived. For example, if the adapting face is male anti-A, the PSE on the relevant test axes (A/B and A/C) should be shifted toward A. For each subject, mean baseline and adapted PSEs for the relevant axes were defined, such that a perceptual shift in the predicted direction results in a reduction in PSE. The mean baseline and adapted PSEs on the relevant axes were calculated and compared with a paired

*t*-test (2-tailed).

*SD*= 0.058). In the adapted condition, this dropped to 0.310 (

*SD*= 0.058). This perceptual shift was highly significant (paired

*t*-test, 2-tailed,

*df*= 12;

*t*= 5.73,

*p*< 0.0001; see Figure 5). These identity aftereffects cannot be explained by low-level visual adaptation since adapting and test faces were presented in different retinotopic locations (see Methods section). Nor can they be explained by a generalized shift in PSE due to exposure to a face other than the mean; since the identity of the adapting male anti-face was counterbalanced across subjects, the predicted direction of perceptual shift due to adaptation is reversed between subjects.

*SD*= 0.39).