Abstract
Decades of work have established the reconstructive nature of memory. Yet these works focus almost exclusively on retrieval. Here we ask whether the brain recruits reconstructive processing -- in a more automatic and fast form -- even at the earliest stage of memory formation, during perceptual encoding. Image memorability -- that some images are systematically better remembered than others -- provides an opportunity to investigate what bridges perception and memory. Existing accounts of memorability solely evaluate bottom-up feature hierarchies optimized for image classification (using standard deep convolutional neural networks [DCNNs]) as the underlying mechanism but fail to consider the role of reconstructive processes during visual encoding, which can support generative functions and compression during memory formation. In computational models, such reconstructive processes are often implemented in non-feedforward modules (e.g., top-down, lateral) and we hypothesize that they should express a distinct temporal signature in behavior. To test this hypothesis, we used two models to capture reconstructive processing during visual encoding: a sparse coding model and a generative adversarial network. In Study 1, in a scene memorability dataset with over 2000 images, we found that images with larger reconstruction error are more memorable (ps<.001), and that reconstruction error captures additional variance in memorability (ps<.001) beyond what can be explained using distinctiveness, a measure derived from the feature space of a DCNN trained for classification. To demonstrate that the variances captured by distinctiveness and reconstruction error result from functionally distinct processes, we ran a pre-registered Study 2 (N=45) using the rapid serial visual presentation (RSVP) paradigm with varying encoding times (34 to 167 ms). We found that images with large reconstruction error benefit more from longer encoding times than images with small reconstruction error, controlling for distinctiveness (all ps<.05). These results reveal reconstruction error as a previously unrecognized source of image memorability.