Abstract
Painters depict materials by utilizing an implicit knowledge of human material perception. To enable studies of this implicit knowledge, we created a database of paintings annotated with bounding boxes of various depicted materials (e.g., fabric, stone, wood, etc.). First, we collected a set of nearly 20K paintings (primarily Old Masters) from online open-access galleries of nine internationally renowned art institutions. For 14 material categories, we asked human annotators from Amazon Mechanical Turk (AMT) to indicate the presence of each material within each painting. On average, we find 6.4 unique materials per painting. Skin and fabric were the most frequently depicted materials, while ceramic and food were the least frequent. Additionally, we calculated material co-occurrence correlations which show that, for example, when skin is present within a painting, food is less likely to be depicted within the same painting and vice-versa (r = -.59). Next, we selected 15 skilled AMT annotators to annotate more than 100k bounding boxes of the 14 materials. This allowed us to identify the spatial location within paintings where materials are depicted. For example, we find that glass is more often depicted within the top-left quadrant of the painting, which could be related to art-historical literature noting that light is often depicted as originating from the top-left (perhaps through a glass window).
In the next step, more detailed material labels will be obtained, for example fabrics will be refined as silk or cotton or wool, etc. This in-depth dataset of material depictions can enable various perceptual, computational and historical analyses that could enable a deeper understanding of material perception and depiction.