Development of the multisensory perception of water in infancy

Abstract

Material perception is facilitated by multisensory interactions that enable us to associate the visual properties of a material with its auditory properties. Such interactions develop during infancy and are assumed to depend on the familiarity of materials. Here, we aimed to pinpoint the age at which infants acquire multisensory interactions for the perception of water, which is a familiar material to them. We presented two side-by-side movies of pouring water and ice while providing the corresponding sounds of water and ice, as well as silence. We found that infants older than 5 months of age looked longer at the water movie when they heard the sound of water. Conversely, they did not look at the ice movie when they heard the sound of ice. These results indicate that at approximately 5 months of age, infants develop multisensory interactions between auditory and visual properties of water, but not of ice. The contrasting results between water and ice suggest that the development of multisensory material perception depends on the frequency of interactions with materials during infancy.

Introduction

Material perception involves multiple senses, such as vision (Adelson, 2001; Motoyoshi, Nishida, Sharan, & Adelson, 2007) and audition (Arnott, Cant, Dutton, & Goodale, 2008) because the material properties of objects carry multisensory information. Previous studies have shown a strong interaction between auditory and visual properties for material perception, enabling humans to associate the visual properties of a material with its auditory properties to judge the category to which the material belongs (Fujisaki, Goda, Motoyoshi, Komatsu, & Nishida, 2014; Fujisaki, Tokita, & Kariya, 2015). During infant development, audiovisual associations for material properties have been observed from the preverbal stage (Bahrick, 1983; Bahrick, 1992; Bahrick, 2001; Ujiie, Yamashita, Fujisaki, Kanazawa, & Yamaguchi, 2018). Bahrick (1983) showed that 4.5-month-old infants can match the visual appearance of a material with the appropriate sound when observing two side-by-side movies of two striking objects played with the authentic sound of the impact. A recent functional near-infrared spectroscopy study observed brain activation for audiovisual material perception in the right temporal hemisphere from the age of 4 months (Ujiie et al., 2018). 

The development of multisensory perception for a material may depend on its familiarity. Ujiie et al. (2018) showed that infants acquire audiovisual associations for the material properties of wood earlier than those for metal. This was demonstrated by comparing the onset of brain mapping for each material between 4 and 8 months. This difference may arise from a lack of familiarity with metal; younger infants may have less experience with metal objects compared to wooden objects in their daily lives. Indeed, using the visual deprivation approach, previous studies have demonstrated that visual experience is necessary for the development of multisensory integration (Putzar, Goerendt, Lange, Rösler, & Röder, 2007; Wallace, Perrault, Hairston, & Stein, 2004). In the monkey brain, after simple long-term visuohaptic experience with materials, the emergence of cortical activation in response to visual-based material perception in the ventral visual pathway has also been reported (Goda, Tachibana, Okazawa, & Komatsu, 2014; Goda, Yokoi, Tachibana, Minamimoto, & Komatsu, 2016). 

Here, we hypothesized that infants acquire audiovisual associations for material properties of water earlier than for any other materials. Through bath time, meal time, and other life events, infants have plenty of experiences with water through multiple senses right from birth. In general, infants are commonly exposed to water through feeding at meal time. Infants have higher fluid requirements (around 150 mL/kg of bodyweight) as compared to adults (approximately 50 mL/kg of body weight) because the total body water composition of infants (75%) is greater than that of adults (50%–60%) (Coe & Williams, 2011). Furthermore, infants are also exposed to water via bathing at least two or three times a week (Lavender et al., 2011). If material familiarity is an important factor, infants may acquire audiovisual association for the material properties of water earlier than 4 months of age, which is when the association for wood as a material develops. 

On the other hand, previous studies on the visual development of water perception indicate that the acquisition of audiovisual association for water emerges later than 5 months of age (Hespos, Ferry, Anderson, Hollenbeck, & Rips, 2016; Hespos, Ferry, & Rips, 2009). Water itself has unique physical properties such as shapelessness and continuous form (Houix, Lemaitre, Misdariis, Susini, & Urdapilleta, 2012; van Assen, & Fleming, 2016). These properties provide motion information that is useful when visually perceiving and categorizing water (Hespos et al., 2009; Hespos et al., 2016; Kawabe, Maruya, Fleming, & Nishida, 2015). During development, infants can discriminate visually between water and other objects from the age of 5 months (Hespos et al., 2009; Hespos et al., 2016). Using a visual habituation paradigm, Hespos and colleagues showed that 5-month-old infants can differentiate between the movements of water and objects such as a polyester (Hespos et al., 2009) or sand (Hespos et al., 2016). These results indicate that infants acquire knowledge of the visual properties of water at least by the age of 5 months; therefore, audiovisual associations for water may also be acquired at later time points. 

This study aimed to investigate the age at which infants acquire audiovisual associations for the material properties of water, which is a more familiar material than wood (Bahrick, 1983; Ujiie et al., 2018). In this study, we used a visual paired-comparison procedure to test whether infants could match the sound of pouring water with a movie showing the same. We presented two side-by-side movies of pouring water and ice under three sound conditions: sound of pouring water, sound of ice, and silence. We considered that water is a material familiar to infants (Coe & Williams, 2011; Lavender et al., 2011), whereas ice is less familiar. The silent condition was a control condition to confirm that the infants had no bias toward the water movie over the ice movie. We expected that if the infants had acquired the association between the visual and auditory properties of water, they would look longer at the movie of pouring water when presented with the water sound as compared to when presented with silence. 

Methods

Participants

All infants were full term at birth and healthy at the time of the experiment. Fifty-four healthy infants participated in the experiment: 18 infants were 3 to 4 months old (12 girls and 6 boys, mean age = 112 days, range = 78–132 days); 18 infants were 5 to 6 months old (10 girls and 8 boys, mean age = 173 days, range = 140–193 days); and 18 infants were 7 to 8 months old (11 girls and 5 boys, mean age = 237 days, range = 203–254 days). Three infants were excluded due to fussiness. An appropriate sample size (n > 16) for each age group was calculated via power analysis (pwr package for R 3.3.0; R Foundation, Vienna, Austria) to detect a Cohen's d of 1.0, statistical power (β) of 0.8, and significance level (α) of 0.05 in a two-tailed, one-sample t-test against chance level. 

This experiment was conducted according to the tenets of the Declaration of Helsinki and approved by the Ethical Committee of Chuo University. Parents gave prior written informed consent for the participation of their children and for publication in an online open-access journal. 

Stimuli

Visual stimuli (480 × 270 pixels) were recorded movies of human hands pouring water or ice from a glass cup into a glass bowl. The auditory stimuli were sounds of pouring each substance. The duration of each stimulus was 4.85 seconds. The congruency of the stimuli was based on the sound of the substance. The experimental task consisted of three conditions: the water sound condition, the ice sound condition, and the silent condition (Figure 1). In the first two conditions, the paired movies were displayed side-by-side while the sound of either pouring water or pouring ice was simultaneously presented. In the silent condition, which was a control, the movie pairs were displayed without any accompanying sounds. 

Figure 1.

 

Combinations of audio and visual stimuli in each condition: (a) control condition (silent), (b) ice sound condition, and (c) water sound condition.

We manipulated temporal information to synchronize the presentation of the visual stimuli for each sound. For the water and ice sound conditions, the auditory information was synchronized with the visual stimuli that were played side by side. Therefore, only the relationship between the type of sound and the type of substance was relevant for audiovisual matching. 

Procedure

The visual stimuli (480 × 270 pixels) were displayed on a 21-inch cathode ray tube (CRT) monitor with a resolution of 1024 × 768 pixels, placed in front of the infant at a distance of 40 cm. The audio stimuli were presented at a sound pressure level of approximately 60 decibels (dB) through two loudspeakers located on the left and right sides of the display. To record the infant's visual behavior, a pinhole camera was placed below the display. 

Each infant was seated on a parent's lap and looked at the stimuli on the monitor without performing any active tasks. The infants were presented with all three conditions. Each condition included two trials, and the stimuli were presented twice per trial. The duration of each trial was 9.7 seconds. The position of the stimuli was reversed in the second trials, although all other conditions remained the same. The order of presentation (left or right) in each condition was counterbalanced for each infant. To attract the infant's attention, a fixation figure was shown in the center of the CRT monitor accompanied by a simple sound prior to each trial. After confirming that the infant was looking at the fixation figure, the researcher started the trial. 

Data analysis

The observer measured the duration for which the infant looked at each visual stimulus based on an offline video movie, henceforth referred to as looking time. The observer, blinded to the lateral positions of the target movie, recorded the looking time for the left or right presentation field by pressing one of two keys corresponding to the relevant field. When the infant looked away from the presentation field, no recording was made. Inter-observer reliability was calculated for the primary observer and a second, trained observer for all conditions and all age groups. Reliabilities for the preference scores were calculated based on the correlation between the looking times rated by both observers in the direction of the infant's fixations. Pearson correlation between the two observers’ ratios revealed a sufficient level of reliability (r = 0.93). 

From these recorded data, we calculated preference scores for the water movie; the preference score was the ratio obtained by dividing the looking time toward the water movie by the looking time toward both movies. A positive value of this score indicated preference for the water movie, and a negative value indicated preference for the ice movie. In subsequent analysis, we conducted a mixed analysis of variance (ANOVA), with age group as a between-participants factor and condition as a within-participants factor, using the water preference scores. We expected a significant difference in this score between the silent and water (or ice) sound condition if the infants appropriately matched the sound with the visual image. 

Results

The mean preference scores for the water movie in each condition for each age group were calculated (Figure 2). To test whether the infant could match the sound of water (or ice) with the appropriate visual image, we conducted a mixed ANOVA with age group as a between-participants factor and condition as a within-participants factor. The mixed ANOVA revealed a significant main effect of conditions [F(2, 102) = 5.61, p < 0.01, η²_partial = 0.09] and a significant interaction between conditions and age groups [F(4, 102) = 3.04, p < = .05, η²_partial = 0.10]. The main effect of age groups was not significant [F(1, 30) = 1.44, not significant]. Analysis of interaction showed simple main effects of conditions in infants 5 to 6 months old [F(2, 102) = 4.13, p < 0.05, η²_partial = 0.35] and infants 7 to 8 months old [F(2, 102) = 6.89, p < 0.01, η²_partial = 0.59]. Multiple t-tests corrected by the Holm method showed that the infants looked longer at the water movie in the water sound condition than in the silent condition in the age groups of infants 5 to 6 months old [t(17) = 3.28, p <.05] and infants 7 to 8 months old [t(17) = 3.27, p < 0.05]. In contrast, there were no differences in preference scores between the ice sound condition and silent condition across all age groups (p > 0.05). 

Figure 2.

 

The infants’ preference scores for the water movie in the three age groups; error bars represent ±1 SEM. Asterisks indicate significance level of the statistical differences; ^*p < 0.05.

Furthermore, to confirm that an infant had no bias toward the water movie in the silent condition, we conducted two-tailed, one-sample t-tests against the chance level (0.5). The t-tests showed that looking times toward the water movie were comparable to those toward the ice movie in the silent condition across all age groups: for infants 3 to 4 months old, t(17) = 0.00003, p = 0.999; for infants 5 to 6 months old, t(17) = 1.11, p = 0.28; and for infants 7 to 8 months old, t(17) = 1.16, p = 0.26. 

Discussion

This study aimed to investigate the age at which infants acquire audiovisual associations for a material property (water) that is relatively familiar to them. Our results showed that infants 5 to 6 months of age and infants 7 to 8 months of age looked longer at the movie of pouring water in the water sound condition than in the silent condition. Conversely, none of the infants looked at the ice movie in the ice sound condition. We also confirmed that there were no differences in preference between the water and ice movies in the silent condition. These results indicate that infants older than 5 months of age can acquire audiovisual associations for water as a material, but not for ice. 

Our findings imply that the development of multisensory water perception is closely linked to the visual development of water perception. We found that the onset of development of multisensory water perception is at approximately 5 months of age, which is consistent with the age of the acquisition of visual water perception (Hespos et al., 2009; Hespos et al., 2016). This indicates that infants can associate the auditory property of water with its visual property after they acquire knowledge of the visual properties of water at 5 months of age. The same interpretation can apply to the difference in timing of acquiring multisensory perception for object materials. For example, infants acquire the sound–object association for the material properties of metal after approximately 6 months of age (Ujiie et al., 2018), which is also when they develop visual perception of metal (e.g., Yang, Kanazawa, Yamaguchi, & Motoyoshi, 2015; Yang, Otsuka, Kanazawa, Yamaguchi, & Motoyoshi, 2011). 

Contrasting results between the water and ice conditions suggest the role of experiences in the development of multisensory material perception. In our study, infants 5 to 6 months old and infants 7 to 8 months old could not match the sound of ice with the movie of pouring ice, but they could do so for water. The null results in the ice condition cannot be due to impaired visual development because infants older than 5 months can discriminate visually between water and ice (Hespos et al., 2009; Hespos et al., 2016). Our results indicate that, as compared to water, infants have less opportunity to interact with ice in daily life and have less familiarity with the material properties of ice. Indeed, crucial roles for experience have been found in the development of our perceptual system (Lewkowicz & Ghazanfar, 2006; Pons, Lewkowicz, Soto-Faraco, & Sebastián-Gallés, 2009; Putzar et al., 2007; Wallace et al., 2004). Several studies have also demonstrated that experiences during infancy carry specific information that can promote and tune our perceptual system to be effective in perceiving environments. This has been shown for tasks such as face perception (Pascalis, de Haan, & Nelson, 2002), voice discrimination (Werker & Tees, 2002), and face-voice association (Lewkowicz & Ghazanfar, 2006; Pons et al., 2009). Conversely, a lack of visual experience impairs the ability to integrate visual and other sensory information (Putzar et al., 2007; Wallace et al., 2004). Based on our study, we propose that infants acquire the association for audiovisual properties of ice later in development, when they typically have more experience with ice. 

We consider that the null results in 3-month-olds are not explained by limitations in their audiovisual integration abilities. Previous studies have clarified that 3-month-old infants can learn to perceive associations between the visual property and the appropriate sound of specific objects (Bahrick, 1988) and between a voice and face (Brookes, Slater, Quinn, Lewkowicz, Hayes, & Brown, 2001). Based on this, our results indicate that 3-month-old infants did not match the visual property of water with the sound due to a lack of experience. Further investigations using the familiarization paradigm would be effective to directly investigate this. This paradigm assesses the audiovisual integration ability in 3-month-old infants (e.g., Bahrick, 1988; Brookes, et al., 2001) by first familiarizing a pair of auditory and visual stimuli and then testing infant preferences. When the infant demonstrates a preference for a novel pair of audiovisual stimuli as opposed to a familiar one, this is interpreted as a marker of audiovisual integration. Using this paradigm, future studies should clarify whether the null results in 3-month-olds are due to their immature audiovisual integration abilities or, rather, a lack of experience. 

Additionally, it remains unclear precisely how much experience infants require with novel materials to learn the multisensory association with the material's properties. It has previously been reported that, in monkeys, exposure to material objects over 2 months shaped the neural representation of the object, leading to perception of its material properties (Goda et al., 2016). Likewise, a similar effect can be assumed in human infants. Exposure to novel material during a certain period can facilitate multisensory association with its material properties. Indeed, one study observed that exposure to foreign speakers over 4 weeks affects the phonetic foreign language learning in 9-month-olds (Kuhl, Tsao, & Liu, 2003). Future studies should clarify the learning process of multisensory material perception through experience by manipulating the duration of exposure to novel materials. 

Conclusions

In summary, this study demonstrated that the audiovisual associations for material properties for water develop at approximately 5 months of age, whereas those for ice are not acquired until after the preverbal stage. Such development might depend on familiarity of a material, based on the infants’ experiences in their daily lives. 

Acknowledgments

The authors thank S. Nishida, PhD (NTT Communication Science Laboratories), I. Dan, PhD (Chuo University), and K. Takahashi, PhD (Chukyo University). We also thank all of the infants and their parents who participated in our experiments. 

Supported by the Grant-in-Aid for Research Activity Start-up program (grant no. 16H07207), Early-Career Scientists (grant no. 19K20650), and Grant-in-Aid for Japan Society for the Promotion of Science Fellowship Program (grant no. 19J00722). It was also supported in part by JST-RISTEX, as well as by a Grant-in-Aid for Scientific Research on Innovative Areas (17H06343, “Construction of the Face-Body Studies in Transcultural Conditions,” and 18H05014, “SHITSUKAN Science and Technology”). 

All authors contributed to the study design. Testing, data collection, and data analysis were performed by YU under the supervision of SK and MKY. Data interpretation was performed by all authors. YU drafted the manuscript, and SK and MKY provided critical revisions. All authors approved the final version of the manuscript for submission. 

Commercial relationships: none. 

Corresponding author: Yuta Ujiie. 

Email: [e-mail]. 

Address: Graduate School of Psychology, Chukyo University, Showa-ku, Nagoya-shi, Aichi, Japan.