Abstract
Deep Convolutional Neural Networks (DCNNs) recognize faces over image/appearance variation (e.g. viewpoint, illumination, expression) while retaining information about this variation (Hill et al., 2019). Here, we examined the extent to which DCNNs encode the 3D shape and surface reflectance properties of a face in the presence of challenging image variability. We generated synthetic face images using parametric models of 3D face shape and surface reflectance. The FLAME model uses a linear shape space to generate faces by varying 3D shape and expression parametrically (Li et al., 2017). The reflectance model was based on the Basel Face Model (Paysan et al., 2009). We generated five face shapes and five reflectance maps to create 25 unique faces by combining all possible pairs of shapes and reflectance maps. The stimulus set (n=1,125) consisted of images of these faces rendered at 9 viewpoints (yaw angle: 0°, 30°, and 60°; pitch angle: -30°, 0°, 30°) with 5 expressions. Each image was processed by a DCNN trained for face identification (ArcFace: Deng et al., 2019) to generate a vector representation. Cosine similarity was calculated between all possible pairs of image representations. Across this set of “identities”, shape and reflectance are confounded, with each identity sharing both its shape and reflectance with many other identities. We asked whether the network's reflectance code is tolerant to changes in viewpoint and expression by splitting the cosine similarity scores into distributions of same-reflectance and different-reflectance pairs. Results showed that the DCNN encodes reflectance over viewpoint and expression variation (Area Under the ROC Curve, AUC=0.66). Analogously, the distributions for same-shape and different-shape pairs yielded an AUC of 0.58, indicating that the shape properties of the face are encoded also. Therefore, DCNNs can separately encode the fundamental dimensions of shape and reflectance in the face, despite wide variations in viewpoint and expression.