Audiovisual short-term influences and aftereffects in motion: Examination across three sets of directional pairings

Abstract

The study of cross-modal influences in perception, particularly between the auditory and visual modalities, has been intensified recently. This paper reports on a comprehensive study of auditory-visual cross-modal influences in motion, including motion aftereffects (MAE). We examined both auditory influences on visual perception and vice versa. Visual motion interactions were examined using three directional pairings or configurations: along the horizontal, vertical, and depth axes. In Experiment 1 we assessed how the simultaneous presence of a strong motion signal in one modality affected the perception of motion in the other modality. To investigate further whether such influences have long-term effects, we tested whether adaptation in one modality alone could produce cross-modal MAEs in Experiment 2. Overall, the pattern of results was similar across all directional pairings, with the strongest cross-modal influences observed in motion along the horizontal axis; this is likely due to the greater co-localization of the two stimuli in this configuration. Although both auditory and visual stimuli affected the other modality when presented simultaneously, significant cross-modally induced aftereffects could only be produced using visual stimuli. However, we did observe vertical visual MAE following adaptation to auditory spectral motion. These results are discussed in terms of current psychophysical and neurophysiological findings concerning the way in which auditory-visual signals are processed.

Introduction

Our perceptual system is bombarded with diverse information arriving from different sensory modalities, yet we achieve an integrated, coherent percept of our environment. The perceptual system exploits spatial and temporal correlations among sensory events in order to maximize our sensitivity to external cues (Alais & Burr 2004a; Stein & Meredith, 1990). The information from one sensory modality may also be weighed more heavily than others in forming an impression of events (Hillis, Ernst, Banks, & Landy, 2002). For example, the sound of an animal in the woods can indicate a potential threat even when it can be neither seen nor localized through sound effectively. Once the target is foveated, however, the visual system provides the most precise spatial resolution. The perceptual system must therefore be adaptive and dynamic. The study of cross-modal influences provides some insight into the complex processes that coordinate and maintain a reliable representation of the environment. 

Much research has shown that for spatial events, when auditory and visual cues are discrepant, the sound can appear to emanate from the visual source. The well-known ventriloquist effect (Bertelson & Radeau, 1981; Thurlow & Jack, 1973) is an excellent example of this phenomenon. We experience this effect in our daily lives when the sound from television loudspeakers appears to derive directly from the actors on the screen. A static auditory stimulus can also appear to move with a visual moving object (Kitajima & Yamashita, 1999; Mateeff, Hohnsbein, & Noack, 1985). 

With regard to dynamic auditory and visual stimuli, a moving light stimulus can induce perceptual shifts in the apparent velocity as well as the direction of a sound source moving along horizontal, vertical, or depth directions in three-dimensional space (Kitajima & Yamashita, 1999). Similarly, the perceived direction of auditory apparent motion is strongly modified by visual apparent motion (Soto-Faraco, Lyon, Gazzaniga, Spence, & Kingstone, 2002; Soto-Faraco, Spence, & Kingstone, 2004). Yet, auditory spatial events can, under some conditions, induce shifts in the perceived motion of visual stimuli. Meyer and Wuerger (2001) manipulated the proportion of dots that moved in a coherent direction while a supra-threshold auditory signal was cross-faded between two loudspeakers located behind the display. They demonstrated a bias in the perceived direction of visual motion that was consistent with the direction of auditory motion. 

Recently, Maeda, Kanai, and Shimojo (2004) have described an illusion in which an auditory stimulus containing no information about motion in space altered the perception of visual motion. A tone that is ascending (or descending) in pitch biases the ambiguous motion of two horizontal superimposed, oppositely moving gratings to be perceived as moving upward (or downward). 

Although there is some evidence that conditions exist for auditory stimuli to alter the perception of simultaneously presented visual motion signals, cross-modally induced aftereffects have been obtained following adaptation to visual but not to auditory stimuli. Aftereffects are considered particularly important for demonstrating the existence of sensory effects because they are thought to arise from changes to neural mechanisms, after they have been exposed for a long time to a stimulus that activates them. They are unlikely to reflect changes in response bias because their direction is opposite to that expected by response bias on the basis of the observed motion direction. 

Kitagawa and Ichihara (2002) have shown that an auditory aftereffect is elicited from adaptation to purely visual motion: after adapting to an expanding/contracting square, a steady sound was perceived as getting softer/louder. The converse, however, did not occur: after adapting to a sound the intensity of which grew (or diminished) over time, a static square was not perceived to be changing in size, thus extending the visual dominance to cross-modally induced aftereffects. In fact, the visually induced auditory loudness aftereffect described by Kitagawa and Ichihara is sufficiently robust that it can be elicited even when subjects are presented simultaneously with both expanding and contracting discs and they direct their attention to one of the competing stimuli. In this case, a significant loudness aftereffect is produced favoring the attended stimulus (Hong & Papathomas, 2006). 

Vroomen and de Gelder (2003) showed that a visual motion cue can influence the strength of the auditory contingent aftereffect first described by Dong, Swindale, and Cynader (1999). Dong et al. observed that, after subjects adapt to a leftward-moving sound rising in pitch and a rightward-moving sound falling a pitch, a stationary sound with rising pitch is perceived as moving rightward and vice versa. Vroomen and de Gelder observed that aftereffects were significantly larger when subjects viewed a small bright square that moved in tandem with the auditory apparent motion stimulus. For incongruent auditory and visual motion, aftereffects were reversed. Thus, aftereffects were contingent on the visual motion stimulus. 

Studies such as those of Kitagawa and Ichihara (2002) and Vroomen and de Gelder (2003) that examine aftereffects provide evidence that multisensory interactions do occur at a low-level sensory stage, rather than simply reflecting changes in high-level decision processes, such as response-related bias (Vroomen, Bertelson, & de Gelder, 2001). Post-adaptation biased responses would favor the direction of the adapting stimulus, rather than the opposite direction of the motion aftereffect (MAE) that subjects report perceiving. 

Relatively few studies have demonstrated a strong influence of auditory motion stimuli on the perception of visual spatial events. This may be due, in part, to a lack of ambiguity in the visual stimuli. The addition of cross-modal cues may not alter visual perception because the visual system already provides stable, unambiguous representation of events. The perceptual system may be predisposed to use cross-modal information primarily to resolve ambiguity in sensory impressions. Indeed, multimodal neurons of the superior colliculus exhibit maximal response enhancement with pairs of weak unimodal stimuli. As unimodal stimuli become more effective, there is a decline in the amount by which bimodal stimuli modify neural responses (Wallace, Wilkinson, & Stein, 1996). Moreover, there is evidence that auditory and visual motion signals are combined best when they are coincident in space and time. For example, Meyer, Wuerger, Röhrbein, and Zetzsche (2005) used an array of loudspeakers and light emitting diodes to produce apparent-motion signals and observed that combined input to lower thresholds of the bimodal signals required spatial co-localization and coincidence. 

As outlined above, there have been numerous studies on how a stimulus in one modality affects the perception of a simultaneous stimulus in another modality, but fewer studies on cross-modal motion aftereffects (MAEs). The present study used a comprehensive set of stimuli and conditions to examine cross-modal influences in auditory and visual motions. We first studied such influences by assessing performance with brief simultaneous stimuli. To investigate further the origins of these influences, we tested cross-modal aftereffects. In all cases, we examined both types of cross-modal influence: auditory-to-visual and visual-to-auditory. For each cross-modal influence type, we examined influences in three configurations:

1.  
   x-axis motion, i.e., left-/right-moving visual and auditory stimuli.
2.  
   z-axis motion, i.e., perceptually approaching/receding visual stimuli paired with loudness-changing sounds that appear to move in depth.
3.  
   y-axis visual motion (up/down) and pitch-changing sounds (we also used vertical auditory motion in one condition).

The pairing of auditory motion across frequency and visual motion across space in this last configuration appears odd at first glance, but it is motivated by Maeda et al.'s ( 2004) report of cross-modal effects of spectral sound motion on simultaneous vertical visual motion. Our selection of this pairing was deliberate to test two not yet tested hypotheses: first, to test whether Maeda et al.'s (2004) short-term effect generates aftereffects; second, to test cross-modal effects in the other direction, from spatial visual to auditory spectral motion. Thus, we have 12 conditions, in which we use comparable stimuli: 2 Influence Types (auditory-to-visual and visual-to-auditory) × 2 modes (simultaneous and aftereffects) × 3 configurations (motion along x, y, and z-axes).

We hypothesized that much of the apparent dominance of visual over auditory information in spatial events arises because of the higher visual spatial resolution. Auditory and visual motion signals may exist that produce a more equitable distribution of relative cross-modal influences. To address this issue, we used visual and auditory motion stimuli for which we could manipulate the degree of ambiguity with respect to motion direction. In much the same way, Alais and Burr (2004a) have manipulated visual spatial ambiguity and concluded that bimodal localization is not captured by visual stimulation but is governed by optimal integration of visual and auditory information. To introduce ambiguity in visual motion, we superimposed two oppositely moving sinusoidal gratings of the same spatial frequency (counterphase flickering motion). Visual signal strength and direction was manipulated by changing the relative contrast of the two components. Auditory motion signal strength along the x-axis was controlled by manipulating the signal amplitude from two laterally placed loudspeakers. Sound intensity was varied to simulate sound motion along the z-axis. Sounds of gliding pitch, as well as cross-fading energy between two vertically placed speakers, were used to accompany visual motion along the y-axis. 

Cross-modal influences were examined using a strong unambiguous motion stimulus in one modality, which we call primary modality, simultaneously paired with a relatively weaker stimulus in the secondary modality that possessed some degree of ambiguity. In one task subjects indicated the motion direction of the weaker motion signal. In a second task subjects indicated whether the motion directions in the two modalities were the same. Both tasks yielded similar results: the stimulus having the strong motion signal altered perceived motion direction regardless of whether it was visual or auditory. To examine cross-modal aftereffects, subjects adapted to strong motion stimuli in the primary modality and subsequently discriminated motion direction in the secondary modality. We found that visual stimuli generated significant cross-modal auditory aftereffects, yet visual aftereffects could not be elicited using spatial auditory motion as adapting stimuli. However, auditory spectral motion did induce visual vertical motion aftereffects. 

Methods

Subjects

Five subjects took part in horizontal and motion-in-depth configurations while eight observers took part in vertical motion configuration in Experiments 1a and 2a, including one of the authors (SS), who subsequently contributed to the design of Experiments 1b and 2b. Six subjects participated in Experiments 1b and 2b. An additional two observers took part for the vertical motion configuration in Experiment 1b. All subjects, including co-author SS, were naïve as to the purpose of the experiments at the time of testing. Experiments 1a and 2a were run concurrently and were followed by Experiments 1b and 2b. They are presented here in conceptual rather than chronological sequence. Subjects had normal hearing and normal, or corrected-to-normal, vision. The study was conducted in compliance with the standards set by the ethics committee at Rutgers University. Subjects gave their informed consent prior to their inclusion in the study and were paid for their participation. 

Apparatus

All stimuli were generated using a Windows-based Dell XPS PC computer and presented on 21-inch CRT monitors (Sony Trinitron for Experiments 1a and 2a, NEC AS120-BK for Experiments 1b and 2b). Screen resolution was 1024 × 768 pixels and the frame refresh rate was 75 Hz. Auditory stimuli were generated by using the front two channels of the Creative Megaworks 550 speaker system. All experiments were conducted in sound-insulated rooms with sound-absorbing properties to minimize echoes and interference from external sounds. Experiments 1a and 2a were conducted in a room the walls of which were draped with heavy sound-absorbing fabric with 150% fullness. Experiments 1b and 2b were conducted in a soundproof booth (Acoustic Systems, Model RE146). 

Stimuli

Stimuli were generated and presented using routines from the PsychToolBox (Brainard, 1997; Pelli, 1997) and MATLAB signal-processing and image-processing toolboxes (Mathworks). Visual stimuli were shown against a uniform gray background on a square aperture with a side of 22 degrees and 48 minutes of visual angle. Mean luminance of all displays was 27.5 cd/m². Luminance measurements were obtained using a Minolta CS-100 photometer. Visual stimuli consisted of either a single moving high-contrast (peak Michelson contrast 92.6%) sinusoidal luminance grating or two superimposed low-contrast (peak Michelson contrast 9.1%) luminance gratings moving in opposite directions. The gratings (spatial frequency 0.3 cpd at the viewing distance of 60 cm, temporal frequency 9.4 Hz) were spatially enveloped by a Gaussian function (σ = 4.45°). 

Two loudspeakers were situated on either side of the visual display at approximately ear level. Auditory stimuli were generated either by varying the intensity of a 550 Hz pure tone in two laterally placed speakers (broadband noise in two vertically placed speakers for one vertical motion condition) or by gliding the pitch of a pure tone from 200 Hz to 2700 Hz (or vice versa) in logarithmic steps. The auditory stimuli had a 20-ms ramp-up time and a 20-ms ramp-down time to avoid onset transients that would generate audible “clicks”. Mean sound intensity was always 75 dbA and was varied at a rate of 10 dbA/sec for the motion-in-depth condition. Measurements were made using a Radio Shack digital sound level meter (Model# 33-2205). 

Procedure

Auditory-visual motion pairings were examined along three directional axes (i.e., three configurations). These are shown in Table 1 and depicted in Figure 1. We varied the (secondary) visual motion strength by changing the relative contrast of two superimposed gratings moving in opposite directions; namely, we used complementary levels of contrast ( C and 1 − C) for the two oppositely moving gratings. We varied the (secondary) auditory motion strength by changing the slope of the intensity (or pitch) per unit time. The method of constant stimuli was used in all experiments (50 trials per motion strength). We tested seven levels of motion signal strength in Experiments 1a and 2a and five in Experiments 1b and 2b (the two extreme motion strengths of Experiments 1a and 2a were not used). 

Table 1

 

Visual and auditory pairings used in experiments.

Motion configuration	Visual stimuli	Associated auditory stimuli
Horizontal motion in frontal plane	Vertical gratings moving leftward/ rightward	Sound energy transferred between two laterally placed speakers
Vertical motion in frontal plane	Horizontal gratings moving up/down	Sound gliding up/down in pitch played from both speakers (Maeda et al., 2004)
Motion in depth, i.e., looming or receding	Concentric gratings that expand/contract	Sound gets louder/softer played from both speakers (Kitagawa & Ichihara, 2002)

Figure 1

 

Visual and auditory pairings used in the experiments. The left panel shows the auditory stimuli while the right panel shows the visual stimuli. The top, middle, and bottom panels show the motion along x-, y-, and z-axes respectively.

Experiment 1. Cross-modal influences—Simultaneous presentation

Visual and auditory stimuli were simultaneously presented for 750 ms on each trial. The primary modality contained a strong unambiguous motion signal. The independent variable was the degree of ambiguity in the secondary signal. 

In Experiment 1a subjects attended only to the secondary modality. They performed a direction discrimination task (two-alternative forced choice) on the secondary signal in the presence of a strong primary cross-modal motion stimulus. For each of the six conditions (2 influence types × 3 configurations), 2 sessions of 350 trials each were run; these were split into 5 blocks of 70 trials (700 total trials). The direction of primary motion was alternated across blocks. Subjects underwent training at the beginning of each session, in which they performed the direction discrimination task, in the absence of the strong cross-modal signal. Subjects had to achieve 80% correct discriminations on a training task that tested a range of “motion strengths” in auditory or visual stimuli before they could move on to the actual experiment. The duration of each session was about 30 minutes. 

In addition to the three configurations defined in Table 1, we also repeated Experiment 1a in the vertical motion configuration with auditory motion in the vertical direction (cross-fading sound energy between two vertically placed speakers) instead of auditory spectral motion. We used broadband noise as the auditory signal because in a pilot study we found the auditory direction discrimination to be extremely difficult with pure tone signals. Five naïve subjects participated in the study. 

In Experiment 1b, we changed the task because we wanted to try another way of observing cross-modal effects. In Experiment 1a, subjects effectively ignored the primary stimulus, because they were asked to attend only to the secondary stimulus. However, such a protocol might reduce the probability of observing cross-modal effects, because subjects were ignoring the very stimulus that was supposed to influence the secondary stimulus. To avoid such a conflict, we asked subjects to attend to both modalities in Experiment 1b. Subjects' task was to attend to the direction of motion of both visual and auditory stimuli and indicate whether they moved in the same or opposite directions. The direction of primary motion was randomized across trials. Reaction times for each response were also measured. Perceived motion direction of the weak motion stimulus was inferred from the direction of the strong motion stimulus and the subjects' response. Each condition was tried in separate sessions, with blocks of 100 trials each (500 total trials). 

Experiment 2—Cross-modally induced motion aftereffects (MAEs)

In this experiment we assessed the magnitude of cross-modally induced MAE in the secondary modality following adaptation to a strong motion signal in the primary modality. Subjects attended to the adapting motion signal for 60 s on the first trial, and 6 seconds thereafter in top-up trials. Figure 2 shows the temporal sequence of a typical block of trials in a session. Subjects were required to perform a simple attentive task on the adapting stimuli to engage their attention: during visual adaptation, motion speed was reduced by a factor of two for a very brief interval (250 ms duration); subjects were instructed to detect such events, to ensure that they paid attention to the primary stimulus. Auditory adapting stimuli were generated by repetitions of a 1-s-long motion signal. One of the 6 repetitions in each top-up adaptation interval was presented at a slightly higher or lower (by 1%) frequency than the others. Subjects were required to attend to these events (speed change in the visual or odd frequency in the auditory adapting signal) and respond at the end of the trial by pressing one of two keys, depending on whether they detected a change or not. The performance on the attentive task was required to be accurate on more than 80% of the trials for the results of the block to be included in the analysis. 

Figure 2

 

The temporal sequence of the first few trials in MAE experiments (Experiment 2).

In Experiment 2a, subjects adapted to a strong motion signal only in the primary modality ( unimodal adaptation). A test signal was presented for 250 ms in the secondary modality, after a 200-ms interstimulus interval. Subjects were asked to perform a direction discrimination task on the test stimulus. Each condition comprised three sessions with blocks of 50 trials (700 total trials). Motion direction was alternated across blocks. A block of training trials similar to Experiment 1a preceded each session. 

Experiment 2b was designed to explore further the nature of cross-modal aftereffect, as follows: One possible explanation for such effects is that adaptation to a strong motion signal in the primary modality affects, during adaptation, neurons in the secondary modality that are tuned to the same direction. We reasoned that if, to the strong primary motion signal, we added an ambiguous-direction motion signal in the secondary modality during adaptation, then the ambiguous signal might be affected and perceived to move in the direction of the primary signal; this, in turn, might activate and adapt neurons in the secondary modality that are tuned to this direction, thus producing stronger aftereffects than under unimodal adaptation. Thus, Experiment 2b was the same as Experiment 2a, except that an ambiguous motion signal in the secondary modality was added to the strong signal in the primary modality during adaptation ( bimodal adaptation). Each condition comprised two sessions with blocks of 50 trials (500 total trials). 

Results

Psychometric functions were obtained by plotting the percentage of reported motion (i.e., rightward, looming, and upward) for each of the 3 configurations and conditions ( x-, z-, and y-axis motion, respectively) as a function of motion strength of the secondary (weaker) modality. Figure 3 illustrates cross-modal influences using psychometric curves from Experiments 1b (left panels) and 2b (right panels), averaged across subjects, for horizontal motion. The auditory rightward motion strength along the abscissa (top panels) is defined as the fraction of the spatial extent of rightward motion. Hence, strength of 1 would mean strong rightward motion, 0.5 would mean stationary sound, and 0 would mean strong leftward motion. Similarly, visual rightward motion strength (bottom panels) is defined by the contrast, C _R, of the rightward moving grating (the leftward moving grating contrast is always 1 − C _R). Therefore, strength of 1 denotes strong rightward motion, 0.5 denotes stationary counterphase flickering gratings, and 0 denotes strong leftward motion. 

Figure 3

 

Psychometric curves averaged across observers. (a) and (c) show results from Experiment 1b. (b) and (d) show results from Experiment 2b. Vision is the primary modality in top panels, while auditory is the primary modality in the bottom panels. Filled squares and solid curve show average observer response when primary modality is moving rightward. Hollow circles and dashed curve show average observer response when primary modality is moving rightward.

To determine the size of cross-modal effects, a Weibull function was fitted to the data for each participant under each experimental condition. The point of subjective equality (PSE) was calculated as the signal strength that was required to obtain the 50% point on the ordinate. The PSEs for the two curves in each panel of Figure 3 are indicated on the panel. It must be noted that these PSEs were obtained for illustration purposes only from the curves that fitted the corresponding data, averaged across subjects. In fact, the PSEs for individual subjects were obtained and then averaged across subjects in Figures 4 and 5. The PSEs for the simultaneous presentation of Experiment 1 are shown in Figure 4. The panels in the left and right columns in Figure 4 plot the results for Experiments 1a and 1b, respectively. The top, middle, and bottom panels show, respectively, results for the left/right, approaching/receding, and up/down configurations. Within each panel on the left and right columns, the left pair of bars shows auditory-to-visual cross-modal influence, while the right pair shows visual-to-auditory cross-modal influence. The dark/white bar in each pair shows the PSE when motion in the primary modality is in the positive/negative direction along the corresponding axis. Figure 5 shows the results of Experiments 2a and 2b on the left and right panels, respectively, using the same conventions as Figure 4. 

Figure 4

 

Average PSEs for various conditions in Experiment 1. (a), (c), and (e) show PSEs for motion along x-, z-, and y-axes, respectively, from Experiment 1a. (b), (d), and (f) show corresponding PSEs from Experiment 1b. Dark/white bars show PSEs when the primary modality moves in the positive/negative direction. The two pairs of bars in each panel correspond to different influence types. The left pair corresponds to auditory (primary) influences on visual (secondary) stimuli, while the right pair corresponds to visual (primary) influences on auditory (secondary) stimuli.

Figure 5

 

Average PSEs for various conditions from Experiment 2. (a), (c), and (e) show results for motion along x-, z-, and y-axes from Experiment 2a. (b), (d), and (f) show corresponding PSEs from Experiment 2b. Dark/white bars show PSEs when the primary modality moves in the positive/negative direction. The two pairs of bars in each panel correspond to different influence types. The left pair corresponds to visual (secondary) MAE following auditory (primary) adaptation while the right pair corresponds to auditory (secondary) MAE following visual (primary) adaptation.

The PSE figures for all experiments and conditions were subjected to a two-way repeated measures analysis of variance (ANOVA) across all 3 configurations. There were two “Direction” conditions for each configuration. This factor refers to the opposite directions of the primary signal within a configuration; a significant effect in the ANOVA test will signify cross-modal influences. There are three “Configuration” conditions (right/left, loom/recede, up/down). This ANOVA was performed separately for each of the two “Influence Types,” namely, whether the primary motion signal was auditory or visual. A second two-way repeated measures ANOVA was performed for each configuration using the factors “Direction” and “Influence Type” to determine whether there was a significant interaction between these factors that would suggest a difference in the strength of cross-modal influences, dependent on the influencing modality. The general belief is that, in the spatial domain, the visual modality has a stronger effect on the auditory modality than vice versa. The second ANOVA was run to test specifically if this was true for our experimental paradigm. 

Experiment 1. Cross-modal influences—Simultaneous presentation

If the motion direction in the primary modality plays a significant role in affecting the perception of a signal in the secondary modality, we expect the white PSE bar in each pair to be larger than the corresponding dark bar. Indeed, the results show that this is exactly what happens in all 12 pairs: the white bar is higher by an average of 10.99% (auditory-to-visual 12.11%, visual-to-auditory 9.87%). This provides evidence that, for Experiments 1a and 1b, when a secondary motion (either auditory or visual) is presented simultaneously with a strong cross-modal motion signal, the motion direction of the secondary signal is perceived to be closer to the direction of the primary motion stimulus. 

In Experiment 1a (attend to the secondary modality only) a two-way repeated measures ANOVA across the three motion configurations revealed a significant effect of motion direction when the primary motion signal was auditory [ F(1,15) = 16.61, p < 0.01]. The same trend was observed when the primary motion signal was visual, with marginal significance [ F(1,15) = 4.19, p = 0.0585]. There was no significant effect of motion configuration and no significant interaction effects. Bonferroni posttests showed a significant effect of motion direction for the vertical motion configuration [ t(7) = 3.385, p < 0.05] when spectral auditory motion was the primary signal. 

When we used vertical auditory motion instead of spectral motion, we found a significant effect of motion direction [ F(1,8) = 10.08, p < 0.05] and a significant interaction between influence type and motion direction [ F(1,8) = 7.18, p < 0.05]. This shows that the effect of visual motion on auditory motion was much stronger than in the opposite direction. A posttest revealed a highly significant effect of motion direction [ t(4) = 4.139, p < 0.01] only when vision was the primary modality. 

In Experiment 1b (attend to both modalities) the ANOVA revealed a highly significant effect of motion direction both when the primary motion signal was auditory [ F(1,17) = 12.63, p < 0.01] and visual [ F(1,17) = 13.51, p < 0.01]. There was a significant interaction between motion configuration and motion direction when vision was the primary modality [ F(2,17) = 6.94, p < 0.01]. Bonferroni posttests showed a significant effect of motion direction for the horizontal motion configuration both when vision [ t(5) = 5.006, p < 0.001] and audition [ t(5) = 2.983, p < 0.05] were the primary modality. 

For Experiment 1b, reaction times were significantly smaller when subjects responded that the directions of motion in the two modalities were the same than when they were different. This was true both when vision and audition were the primary modality: [ F(1,17) = 14.69, p < 0.01] and [ F(1,17) = 15.96, p < 0.001], respectively. 

A second two-way ANOVA performed separately for each configuration revealed that there were no main effects of Influence Type or significant interaction effects between Influence Type and Direction. Only Direction was significant or tended toward significance for all configurations. For horizontal motion, [ F(1,8) = 2.67 p = 0.1447] for Experiment 1a, and [ F(1,10) = 9.7, p < 0.05] for Experiment 1b. For the looming-receding condition, [ F(1,8) = 5.87, p < 0.05] for Experiment 1a and [ F(1,10) = 7.0, p < 0.05] for Experiment 1b. For vertical motion, [ F(1,14) = 4.81, p < 0.05] for Experiment 1a and [ F(1,14) = 8.45 p < 0.05] for Experiment 1b. 

Overall, the pattern of results was very similar across the different tasks of Experiments 1a and 1b: the main effect of direction was either statistically significant or tended toward significance for all auditory/visual configurations (it is worth mentioning that the pair-wise differences of averages in Figure 4 are in the expected direction in all 12 cases, even if they do not always reach statistical significance). This means that the subject's task did not significantly influence perception. The results suggest that the motion direction of the primary modality biased the perceived direction of the motion signal in the secondary modality regardless of whether it was auditory or visual. 

Experiment 2—Cross-modal visual/auditory motion aftereffects (MAEs)

The expected pattern within each pair is the opposite to that of Experiment 1. Namely, if the primary modality influences the MAE of the secondary modality, we expect the white PSE bar in each pair of Figure 5 to be smaller than the corresponding dark bar. This happens for all the three motion configurations only for the visual-to-auditory influence type. Significant auditory motion aftereffects following visual adaptation were observed for both “unimodal” and “bimodal” adaptation conditions. For example, after adapting to rightward visual motion, observers were more likely to indicate that a stationary sound was moving leftward and vice versa. Significant visual motion aftereffects were observed following adaptation to auditory spectral motion. Conversely, no visual motion aftereffects were observed following adaptation to auditory spatial motion for both horizontal motion and motion-in-depth configurations. 

For Experiment 2a (unimodal adaptation) the ANOVA showed a highly significant effect of motion direction [ F(1,15) = 14.33, p < 0.01] for the auditory MAE following visual adaptation. Neither motion configuration nor the interaction term was significant. 

For Experiment 2b (bimodal adaptation), contrary to our expectations, cross-modal aftereffects were not significantly stronger than for Experiment 2a. Apparently, the presence of an ambiguous motion signal during adaptation did not alter neural adaptation in the secondary modality significantly. For this experiment, the ANOVA showed a highly significant effect of motion direction [ F(1,15) = 9.05, p < 0.01] along with a significant interaction between configuration and motion direction [ F(2,15) = 5.24, p < 0.05] for auditory MAE following visual adaptation. Motion configuration was not significant. The Bonferroni posttests showed a very significant effect of motion direction [ t(5) = 4.37, p < 0.01] for the horizontal motion configuration, but the other two configurations were not significant. 

The second ANOVA for each configuration revealed significant visual-to-auditory effects in all three configurations. For the horizontal motion condition there were significant effects of Direction [ F(1,10) = 7.85, p < 0.05] for Experiment 2b. The same pattern was obtained in Experiment 2a, with marginal significance for direction [ F(1,8) = 4.01, p = 0.08]. A paired t-test indicated that adaptation produced significant effects in Experiment 2b [ t(5) = 2.726, p < 0.05], or marginally significant effects in Experiment 2a [ t(4) = 2.081, p = 0.053], but only when the adapting modality was visual. 

For the looming-receding configuration, there was a significant main effect of Direction only for Experiment 2a [ F(1,8) = 6.02, p < 0.05]. A paired t-test indicated that adaptation produced a significant shift in the PSE only when the adapting modality was visual [ t(4) = 2.94, p < 0.01]. 

For the vertical motion configuration with spectral auditory motion, there was a significant main effect of Direction only in Experiment 2a [ F(1,14) = 7.45, p < 0.05]. A paired t-test indicated that adaptation produced a significant shift in the PSE both when the adapting modality was visual [ t(7) = 1.985, p < 0.05] and when the adapting modality involved auditory spectral motion [ t(7) = 2.035, p < 0.05]. 

The results of Experiments 2a and 2b followed a similar trend. The presence of a weak cross-modal signal during adaptation (Experiment 2b) did not alter the pattern but altered the magnitude of the aftereffects. There were cross-modal auditory MAEs following visual adaptation for motion along the x-, y-, and z-axes. We also observed visual MAE following adaptation to auditory spectral motion ( y-axis). 

Overall, these results strongly suggest that although both auditory and visual strong motion signals can produce significant shifts in the PSE when these cross-modal signals are presented simultaneously, cross-modal MAE occur only in one direction: only adaptation to visual motion stimuli influences auditory aftereffects, not vice versa, with the exception of the vertical motion configuration, where we also found visual MAE following auditory spectral motion adaptation. 

General discussion

In Experiment 1 we examined whether a strong motion signal in the primary modality would alter the perception of motion direction in the secondary modality. The high degree of similarity of results across the two tasks, one in which the primary signal was ignored and another in which it had to be attended to, suggests that the perceptual changes induced by strong motion signals are not affected much by attention. It is noteworthy that the stronger modality always biased perceived direction of the weaker modality regardless of whether it was auditory or visual. This was somewhat surprising because of the many studies that suggest visual dominance in the perception of spatial events (e.g., Kitajima & Yamashita, 1999; Mateeff et al., 1985; Soto-Faraco et al., 2002, 2004). 

The cross-modal influences for simultaneously presented auditory and visual stimuli tended to be stronger for the horizontal motion configuration. Perhaps this occurred because in that condition signals were localizable on the same plane along the same axis. The motion-in-depth and vertical motion conditions involved changes in pitch and loudness, respectively, to simulate changes along the y- and z-axes. It is noteworthy, however, that this result appears to contrast with the findings of Kitajima and Yamashita (1999). They examined perceptual distortion of the direction of a sound stimulus when it deviated from the tracks of a light stimulus moving along a horizontal, vertical, or depth direction. Although dynamic capture was evident for all three orientations, it appeared least strongly in the horizontal orientation. The authors suggested that the subjects might have tended to base their judgments of auditory movement more on the direction of movements of the light source for the vertical and motion-in-depth orientations because auditory localization was more difficult to discern for these conditions. We observed that cross-modal influences tended to be stronger for the condition with the least ambiguity in the auditory motion. These differences are likely due to the choice of stimuli in the two studies. Kitajima and Yamashita used a sound that consisted of a noise stimulus that physically moved in space behind a screen, and followed the full range of movement in that plane. We would expect to find a small influence of visual on auditory perception for the condition in which auditory localization was best. Our auditory and visual stimuli varied in signal strength and were constructed such that each could potentially induce cross-modal dynamic capture of the secondary modality. Moreover, our stimuli in the secondary modality had a comparable subjective range of strengths in all three directions (as shown by the comparable range of performances observed in all directions) unlike the stimuli used by Kitajima and Yamashita. In their study, the performance for the horizontal direction reached saturation in the absence of visual motion, while in the vertical motion and motion-in-depth conditions the performance never reached saturation. This indicates a stronger subjective strength in the horizontal direction, which would explain the weaker observed interactions in the horizontal direction. In fact, when we used spatial vertical motion rather than spectral motion in the auditory stimuli, we observed a strong effect of the visual modality on the auditory modality (stronger than the horizontal motion configuration) but the auditory modality failed to significantly influence the visual modality, even though there was a tendency. This apparent imbalance can be explained by the fact that, under our experimental conditions, auditory vertical motion was not as discriminable as auditory horizontal motion, even when we used broadband noise instead of a pure tone for vertical motion. This is because humans are more sensitive to horizontal motion than vertical motion in the auditory domain as shown by the minimum audible movement angle threshold, which is around 4.2° in the horizontal plane and around 15.3° in the vertical plane for broadband noise (Grantham, Hornsby, & Erpenbeck, 2003). 

Maeda et al. (2004) have shown that when an auditory signal was rising or falling in pitch, subjects tended to report upward or downward motions, respectively, for an ambiguously moving counterphase flickering stimulus. We were therefore interested to determine whether this audiovisual pairing was a special case for which the influence of auditory information on visual perception was particularly strong. In fact, auditory stimuli had a stronger effect on visual stimuli than the other way around in both Experiments 1a and 1b. Indeed, the vertical configuration was the only one in which adaptation to a changing pitch sound produced a vertical visual aftereffect. 

An important question that arises is whether the cross-modal influences that were observed in Experiments 1a and 1b reflect auditory-visual signal integration or shifts in response bias. Meyer et al. (2005) examined the conditions under which auditory and visual cues combine to generate lower thresholds for motion detection. Using a 180° arc of loudspeakers and light-emitting diodes that could be independently manipulated, the authors demonstrated that low-level integration of auditory and visual motion signals occurs only when the cues are matched in both position and speed. A study by Alais and Burr (2004b) yielded similar findings: using a dynamic random-dot stimulus varying in motion coherence and a temporally modified stereo noise source, they showed that auditory-visual motion detection thresholds were explainable by the statistical advantage of the combination of signals (i.e., probability summation) and not by linear summation. Yet, there is some very recent evidence that disparate auditory and visual motion signals may induce audiovisual integration in early motion areas. Alink, Singer, and Mucki (2008) demonstrated, using fMRI, that in comparison to non-conflicting trials, capture of auditory motion by vision was associated with an activation shift from auditory to visual motion cortex. More specifically, their research showed that cross-modal dynamic capture reduces activation in auditory motion areas, while increasing activation in particular visual motion areas. Their study is the first to demonstrate neuronal correlates of cross-modal dynamic capture. 

To optimize the chance of obtaining significant effects of auditory stimuli on the perception of simultaneous visual stimuli, we used visual stimuli with a certain degree of ambiguity, thus increasing the probability for such effects to be manifested. Ernst and Banks (2002) had demonstrated, using haptic and visual cues, that the weight of a signal in a particular modality is inversely related to its variability. Alais and Burr (2004a) have shown that this basic principle holds true for spatial location judgments. They pointed out that visual spatial events usually seem to dominate or capture auditory events because they generally have associated with them the lowest amount of spatial uncertainty. The now widely held view is that information from multiple sensory modalities is combined in a statistically optimal fashion (e.g., Alais & Burr, 2004a; Ernst & Banks, 2002; Hillis et al., 2002; Roach, Heron, & McGraw, 2006; Shams, Ma, & Beierholm, 2005; see Ernst & Bülthoff, 2004 for review). 

In Experiment 2 we examined cross-modal influences in aftereffects. Statistical analyses of the data indicated that cross-modally induced aftereffects could be produced by visual but not auditory spatial motion signals. This pattern of results was similar for visual motion along the x-, y-, and z-axes. However, auditory spectral motion could produce visual MAE only in the vertical motion configuration. This result provides further evidence that the visual illusion discovered by Maeda et al. (2004) is perceptual rather than cognitive in origin. 

Our results, in general, are in agreement with those of Kitagawa and Ichihara (2002), who showed that a significant cross-modally induced loudness aftereffect could be produced following adaptation to a disc changing in depth. The magnitude of the effect observed by these authors was stronger than that obtained in our experiment. This was likely due to differences in the adapting visual stimuli. They used a square that changed in size, thus eliciting a strong percept of an approaching stimulus, moving in depth, which is closely coupled to a sound of increasing intensity. On the other hand, we used a concentric grating stimulus that moved radially outward (we used gratings to maintain consistency with the other two configurations in our study). However, our stimuli gave rise to two competing percepts that may have spontaneously alternated during adaptation: looming in depth (approaching) or expanding outward on a frontoparallel plane. It is possible that the expanding percept decreased the effect that the looming percept would exert on the auditory aftereffect. This reasoning is supported by the results of Masuda, Wada, Kitagawa, and Noguchi (2002). They used a stimulus composed of dots spreading outwards from the center that elicit two competing percepts: looming and expanding (on a frontoparallel plane). They reported that the looming or expanding percepts were enhanced when accompanied by a sound of increasing or decreasing intensity, respectively, thus indicating a weak association between expanding visual patterns and sounds of increasing intensity. In addition, the overall size of our grating stimulus remained constant, unlike Kitagawa and Ichihara's square. Changes in size would produce a more robust sensation of looming, perhaps leading to greater changes in mechanisms involved in this process, thus explaining why our effects were smaller than those of Kitagawa and Ichihara (2002). 

Kitagawa and Ichihara (2002) concluded that auditory adaptation does not influence visual aftereffects. It is possible that, contrary to Kitagawa and Ichihara's conclusion, auditory adaptation does influence visual aftereffects; it is just that the magnitude of the influence is small and that their procedure and stimuli were not adequately sensitive to measure such an influence. Our approach in Experiment 2 was to replace the unambiguous square in their aftereffect test stimulus by two competing gratings moving in opposite directions; our ambiguous test stimulus was designed to increase the probability for such aftereffects. It turned out that such cross modal influences for spatial auditory motion along x- and z-axes were not observed even with the ambiguous test stimulus, thus reinforcing Kitagawa and Ichihara's conclusion. 

Conclusions

The most interesting finding of the present paper is that both strong auditory and visual motion stimuli equivalently altered the perception of ambiguous, simultaneously presented cross-modal stimuli. Yet, significant cross-modal aftereffects could only be generated on auditory stimuli following visual adaptation, with the exception of induced visual MAE following adaptation to auditory spectral motion. 

Studies of cross-modal aftereffects are important because they provide evidence that bimodal signals alter perceptions at a neural level and do not simply alter response biases, because the aftereffect direction is opposite to that of the adaptation signal. Aftereffects are thought to arise from changes in neural mechanisms induced via extended stimulus exposure. To date, there have been no reports of cross-modally induced auditory-to-visual aftereffects. The visual vertical MAE induced by adaptation to auditory spectral motion found in the current study is the first instance of cross-modally induced auditory-to-visual MAE. Further experiments (neurophysiological/imaging) are needed to explore the possible brain areas and mechanisms involved. 

Kitagawa and Ichihara (2002) suggested that the visually induced auditory aftereffect may have been due to a multimodal processes combining auditory and visual signals and projecting back, perhaps through feedback connections from higher multimodal areas (Calvert, Campbell, & Brammer, 2000; Driver & Spence, 2000) to the auditory motion-in-depth process. One possibility is that neurons involved in visual motion perception are more resistant to influences from multimodal areas of the brain. A second possibility is that auditory-induced visual aftereffects may be observable under appropriate circumstances. For example, strong continuous auditory motion, based on the physical motion of the sound source across space, might possibly produce measurable visual aftereffects. To carry this point further, it is possible that only ecologically valid signals are bimodally integrated. Wuerger, Hofbauer, and Meyer (2003) have addressed this issue. Their visual motion stimulus was a random-dot kinematogram and the auditory motion signal was white noise cross-faded between two loudspeakers. Motion coherence levels were manipulated by varying the proportion of similarly moving dots and the signal-to-noise ratio of the acoustic signal. They found that the increase in sensitivity to motion in terms of direction and speed could be explained by probability summation. They concluded that the integration of signals appeared to be taking place at a post-decision stage and was not limited to ecologically valid motion signals. However, MAEs are thought to arise from sensory processes. As such, it remains possible that only physical motion of a sound source can elicit visual MAEs. Further, in our experiments, auditory and visual stimuli were constructed to be fairly similar in terms of relative motion strength. Perhaps significant auditory-visual MAEs may be produced with more powerful, robust acoustic signals (e.g., well-localized noise sources). These questions remain to be examined. 

Supplementary Materials

Supplementary Figure 1 - Supplementary Figure 1 

Supplementary Figure 1. Same as Figure 6, except that analyses are shown for each of the 10 participants, and predictions are not shown. Despite some differences in amplitudes and signal-to-noise ratios, participants are remarkably consistent with each other. See Figure 6 caption for details. 

Acknowledgments

This research was supported by NEI/NIH Research Grant R01 EY 013758-01 to Thomas Papathomas. Portions of this paper were presented at the Annual Meeting of the Vision Sciences Society, 2007, Sarasota, Florida. 

Commercial relationships: none. 

Corresponding author: Thomas V. Papathomas. 

Email: [e-mail]. 

Address: Center for Cognitive Science, Rutgers University, 152 Frelinghuysen Road, Piscataway, New Jersey 08854, USA.