Comparing human and deep convolutional neural network face-matching performance on disguised face images

Abstract

People perform poorly on face-matching tasks that involve unfamiliar identities. Disguise further impairs performance, even for familiar identities, which are usually recognized robustly across image variation (Noyes & Jenkins, 2016). Face recognition algorithms based on deep learning and convolutional neural networks (DCNNs) now perform surprisingly well across image variation, but have not been tested with disguised faces. Here we directly compare DCNN accuracy with human accuracy (Noyes, 2016) for identifying disguised and un-disguised faces. Using the features from the top-level "compact layer" of a recent DCNN (Chen, 2016), we generated representations for un-disguised and disguised images of the same faces from the FAÇADE database (Noyes, 2016). To determine whether images would cluster by their true identity, hierarchical agglomerative clustering was applied to the DCNN face representations. This clustering was compared to human identity-matching performance for the same images. Humans and the DCNN were similarly impaired by disguise when the comparison images were of the same identity (disguise involved evading identity). For different-identity trials, human performance dropped with disguise, but the DCNN performed at the same level on un-disguised and disguised (impersonation) comparisons. Next, we compared performance for only disguised images. DCNN accuracy was surprisingly similar to that of humans for evasion faces - with machine accuracy of 61.54% and average human accuracy of 60.38%. For impersonation of a similar looking person, machine accuracy was 84.62% and human accuracy was 82.18%. However, machines performed more accurately than humans (96.15% > 89.62%) at matching faces disguised to impersonate a very different looking individual. Notably, familiar viewers from Noyes (2016) far outperformed machines and unfamiliar viewers on disguised faces. These findings provide insight into the current performance level of DCNNs in comparison to humans for identifying disguised and undisguised faces.

Meeting abstract presented at VSS 2017