Abstract
Telling mirror from glass is highly challenging because both materials derive their appearance from their surroundings. Despite this, humans readily discriminate them, even when colour and luminance distributions are matched. To test how different visual cues contribute to this ability, we trained classifiers to discriminate renderings based on a range of features, and compared responses to human mirror/glass classifications on an image-by-image basis. We created over 750,000 renderings with either ideal mirror or ideal refractive materials, varying the shape, illumination and viewpoint. Then, three classifiers were defined by features based on simple pixel histograms ('Simple'), Portilla-Simoncelli texture statistics ('PS'), and three-layer convolutional neural networks ('CNN'). For randomly selected renderings, humans and all three classifiers performed well. Such high performance levels makes it hard to determine which cues the visual system uses, so to distinguish more precisely between classifiers, we selected a smaller subset of images for which the classifiers' responses were inconsistent, or consistently incorrect, forcing accuracy of the classifiers near to chance level. However, fifteen human observers judged those stimuli with 85% accuracy, suggesting humans use additional cues. The key challenge is to predict both successes and failures of human perception, so we then used Generative Adversarial Networks trained on renderings to create a new stimulus set that uniformly spanned the range from highly diagnostic to highly ambiguous. We then used Bayesian hyper-parameter search to identify CNN architectures that, when trained on standard renderings, also correlate highly with humans on these more ambiguous images. The resulting networks and images reveal many novel cues: e.g., mirrored surfaces tend to exhibit smooth, saturated colour gradients, while glass images have distinctive bright, low-saturation fringes. These insights allow us to create novel illusions.
Meeting abstract presented at VSS 2018