Abstract
Although most researchers agree that language and vision are domain specific each with different representational primitives, it is also true that these systems interact allowing us, e.g., to talk about what we see. What is the mechanism, development and consequence of an interaction between language and vision? We describe one language-vision interaction and lay out a hypothesis and some evidence to explain the underlying mechanism. The visual system sometime fails to encode or bind relevant features of an object. In several experiments we asked whether language could help children and adults bind and maintain visual properties that are difficult to retain in visual memory. In a delayed-matching task, participants were shown squares that were split in half by color (e.g., a square with red-left, green-right) and after a delay, they were shown the target replica, the target reflection (e.g., red-right, green-left), and another distracter. We found that providing specific directional terms to children accompanying the target enhanced their ability to find target matches, avoiding foils. Non-linguistic attentional manipulations did not show these effects, nor did linguistic instructions that did not include the directional terms. Finally, when adults are given the same task as the children in one of three conditions (No-Shadow, Verbal-Shadow, and Rhythm-Shadow), they performed significantly worse in the Verbal-Shadow condition compared to the other shadow conditions. Language appears to play a crucial role in maintaining feature conjunction in memory for both adults and children. We propose the Asymmetric Recoding Hypothesis which states that given an object without an inherent figure and ground, language forces the creation of an asymmetry between the parts, and forces the encoding of a directional relationship between the parts. We argue that given visual and linguistic information, a hybrid representation is formed that combines crucial details from each source.
This work is supported by grant FY12-04-46 from the March of Dimes Birth Defects Foundation and NINDS RO1-050876 to B.L. and Grant number F31HD056629 from the NICHHD to B.D.