A cyclopean visual saltation illusion reveals perceptual grouping in three-dimensional space

Abstract

When a number of visual elements are presented briefly one after the other at two separate locations, mislocalization occurs with elements perceived “jumping” evenly across the space between locations. This is the visual saltation illusion. We investigated whether saltation occurs in three-dimensional (3D) space. In separate experiments, observers judged the perceived positions of the first, second, and last elements for a sequence in which the first two elements were presented at one location, and the third was presented at a second location. If saltation occurred, only the second element was mislocalized to a position between the first and second locations. In Experiment 1, we observed 3D saltation, but only for conditions in which the stimulus was located away from the point of fixation. This effect was also observed when the two locations in depth have no lateral 2D separation (Experiment 2). In Experiment 3, we showed that a locally generated motion aftereffect between the two locations distorts the perceived position in depth of only the second element, which perceptually overlaps with the adapted region. Our results demonstrate the appearance of 3D saltation, and that the illusion represents a process in which higher cortical areas feed back to activate lower level inputs to achieve 3D perceptual “filling in.”

Introduction

One of the goals of the visual system is to extract from visual scenes information that can be used for visually guided behavior (Marr, 1982). However, the visual scene is dynamic and complex, containing both useful and spurious information, which makes implementing this task quite difficult. To make sense of the visual world, the visual system must organize local elements into meaningful units that can be efficiently processed and selectively attended to. This is the process of “perceptual grouping.” Understanding the implementation of perceptual grouping has been a long-standing focus of research; categorization of the mechanisms of perceptual grouping was the project of the early 20th century Gestalt school. The Gestalt principles of good continuation, similarity, smoothness, temporal synchrony, and closure, which outline the conditions under which local elements are associated, have been used to provide robust accounts of the neural computations underlying key visual tasks such as contour integration (e.g., Field, Hayes, & Hess, 1993) and visual completion (such as in Kanisza figures, e.g., Li, Khuu, & Hayes, 2009). 

Perceptual grouping is particularly pertinent to the perception of image motion. Motion can be inferred from the perceptual grouping of stimuli that are briefly presented at separate spatial locations. The classic case is that of “stroboscopic” or “apparent” motion: when two stationary visual stimuli, separated by a fixed spatial distance, are viewed in rapid alternating succession, an illusion of movement is produced. The sensation is that of a single stimulus traversing the shortest path between the two elements. This illusory visual phenomenon arises not from the physical displacement of an object over time, but through a process of grouping (mediated by attentive tracking, see, e.g., Horowitz & Treisman, 1994) and interpolation in which visual motion is inferred from the spatiotemporal characteristics of inducing elements (Cavanagh & Mather, 1989; Kolers, 1963, 1972; Wertheimer, 1912). The computation of apparent motion is reflective of a perceptual filling-in process in which a sensation of movement is generated in the non-physically stimulated space between the two flashed stimuli (e.g., Clatworthy & Frisby, 1973; Kolers, 1963; Yantis & Nakama, 1998). The Ternus display elicits a similar phenomenon. This is a bistable apparent motion stimulus generated by briefly presenting two elements to one side, followed by another two elements to the other side. Perception of either element motion or group motion is possible, depending on conditions under which spatial or temporal grouping is optimal (Kramer & Yantis, 1997; Patterson, Hart, & Nowak, 1991; Wallace & Scott-Samuel, 2007). 

Under appropriate stimulus conditions, the perceptual grouping of briefly presented objects can produce substantial and systematic mislocalization of elements from their veridical position between at least two points of stimulation, generating the perception of motion. The visual “saltation” illusion is elicited when multiple stimuli are presented first to one location, then to another, in regular and rapid succession. When the stimulus is viewed in the periphery, rather than being perceived one after the other at their veridical locations, stimuli are perceived as traveling in equidistant steps across the non-stimulated space between the two locations, up to as much as 5° to 10° of visual angle (Geldard, 1976; Khuu, Kidd, & Errington, 2010. Play and loop Movie Clip 1 for a demonstration of this illusion). As mentioned, Geldard (1976) noted that visual saltation is most apparent when the stimulus is presented in the periphery, thus a requirement for the illusion is uncertainty in the apparent location of briefly presented objects brought about by coarse spatial acuity in the periphery. In a systematic study, Moradi and Shimojo (2004) examined the ability of observers to discriminate between the percepts of apparent motion (generated by presenting two stimuli to separate locations), illusory saltation, and real saltation (i.e., a sequence in which stimuli do actually originate from sequential, equidistant, locations across the area between the first and last locations) as a function of retinal eccentricity. They report that while both real and illusory saltations are differentiable at small eccentricities, and that both percepts were clearly different from apparent motion for all eccentricities and interstimulus intervals (ISI), real and illusory saltations are only indistinguishable at large eccentricities (approximately 12–16 degrees). This result indicates that, in the periphery, the percepts of real and illusory saltatory motions are comparable. 

Shore, Hall, and Klein (1998) proposed to interpret saltation in terms of the Gestalt principle of grouping. They suggested that, where stimulus position is made ambiguous, the sensory system assumes that the rapidly presented stimuli arose from a single source moving from one spatial location to another. The earliest accounts of saltation showed that generation of the illusion is dependent on ISI (e.g., Geldard & Sherrick, 1972). These studies showed that the illusion is most compelling at short ISIs, while longer ISIs result in veridical localization. Of course, what defines a “short” or “long” ISIs depends on the sensory system in question (saltation is observed for audition and touch, as well as vision: Geldard, 1975), but the key point is that, when multiple stimuli are presented rapidly, their locations cannot be individuated, and the sensory system might well resolve their positions through perceptual grouping and filling in. 

The available evidence from visual saltation research consistently supports Shore et al.'s (1998) proposition. First, saltation can be elicited through dichoptic presentation of elements, suggesting a cortical origin for this illusion (Geldard, 1976). Second, illusory saltation can be produced across the blind spot (i.e., stimuli are presented at either side of the blind spot but are perceived as traveling across it), making it likely that the percept arises from a filling-in process (Lockhead, Johnson, & Gold, 1980). Third, transformational changes in form accompany the saltation illusion. For example, if elements at the first and second locations are differently colored, mislocalized elements are seen as a mixture of the two colors (Geldard, 1982). Finally, Khuu et al. (2010) recently showed that motion adaptation affects the path of saltation when adaptation is confined to regions overlapping with the perceived location of elements, but not necessarily their physical positions. Clearly then, saltation must be the result of a high-level interpretation, most likely that of grouping and filling in, and cannot be accounted for solely by low-level spatiotemporal mechanisms. Exactly what mechanisms might underlie grouping, specifically in terms of saltation, is given further consideration in the General discussion section. 

While saltation is reminiscent of well-known spatiotemporal apparent motion phenomena such as Phi, Tau, and Kappa (Brigner, 1984; Geldard, 1982; Geldard & Sherrick, 1986; Lockhead et al., 1980; Wiemer, Spengler, Joublin, Stagge, & Wacquant, 2000), it is phenomenologically different in a number of ways. First, while apparent motion is the percept of an object continuously moving between two stimulated locations in a single continuous step, in saltation, elements are systematically mislocalized, being perceived as originating from distinct equidistant points within the unstimulated space between the two locations. Second, whereas apparent motion stimuli minimally require only one stimulus presentation at each location, saltation can only arise with repeated stimulation to at least one location (therefore, a minimum of three elements needs to be presented; Geldard & Sherrick, 1986; Phillips & Hall, 2001). Third, as mentioned, the saltation illusion only arises with peripheral presentation (of greater than approximately 10 deg), while apparent motion is observed in central as well as peripheral vision (Kolers, 1972). Lastly, the saltation illusion is dependent on steady fixation; eye movements from the fixation point to the stimulus breaks down the percept (as the stimulus is brought into central vision). This is not necessarily the case with apparent motion (Koler & von Grünau, 1977). It would therefore seem that apparent motion saltation are different percepts, though the underlying process for grouping may be similar (see General discussion section). 

The majority of experimentation into perceptual grouping in motion phenomena has been conducted for stimuli presented in and traversing the two-dimensional (2D) image plane. However, to gain a more complete understanding of the mechanisms underlying perceptual grouping, it is vital to also consider whether, and to what extent, perceptual grouping leading to the perception of motion occurs for stimuli distributed in depth. That is, for spatiotemporal stimuli presented at different depths, it is important to know whether grouping occurs to provide a sensation of movement in depth, and if it does, the conditions under which this potential percept occurs. It is worthwhile noting that the operations used to perceptually group 2D stimuli do not necessarily apply to the perception of objects in depth. The processing of 2D stimuli can be facilitated through direct coding of the retinal representation of the image, with motion registered by extended integration and grouping from detectors that sample information from successive points on the retinal image. However, depth information is not explicitly encoded via a projection surface of the human visual system in the third dimension. Rather, it must be indirectly inferred from a host of monocular and binocular depth cues. At present, little is known about whether and how depth cues facilitate the perception of motion inferred from the perceptual grouping of spatiotemporal stimuli. The significance of this issue lies in the fact that the visual world is 3D in structure and moving objects frequently traverse both 2D and 3D spaces simultaneously. Understanding how the perceived position in depth of objects inferred from perceptual grouping (leading to saltation) is determined will provide a more comprehensive description of how the visual system is able to represent information in the 3D visual environment. 

Some previous studies have reported perceptual grouping leading to apparent motion in depth (e.g., Green & Odom, 1986; Phinney, Wilson, Hayes, Peters, & Patterson, 1994; Regan & Beverley, 1973). For example, Regan and Beverley (1973) used an apparent motion stimulus in which the two stimuli differed in stereoscopic disparity, generating the percept of alternation in depth. Additionally, it has been shown that stereoscopic apparent motion can be generated using dynamic random-dot stereograms to produce a cyclopean stimulus without monocular cues (e.g., Julesz & Bosche, 1966; Norcia & Tyler, 1984). However, while apparent motion is observed with these configurations, the generated sensation of motion traversing the space between the stimuli is akin to motion blur. Because of this, the apparent motion stimulus limits the study of three-dimensional (3D) grouping because it does not allow careful and precise quantification of the effect beyond simple judgments about the quality (e.g., smoothness) of the motion percept. In this regard, visual saltation is a more useful stimulus: if perceptual grouping occurs consistently with a single object traversing the space between the two points of stimulation, it should be immediately discernible as mislocalization of intermediate elements to intermediate positions in depth. Accordingly, through measurement of the perceived position in depth of mislocalized elements, and the degree to which stimulus conditions affect perceived position in depth, the parameters of perceptual grouping can be carefully quantified. While only a handful of studies have established the existence of apparent motion in depth, no studies have established whether briefly presented elements at different depths lead to the perception of visual saltation in depth. To carefully examine this issue was the goal of the present study. 

With this in mind, a cyclopean version of the visual saltation illusion was used in the present study to investigate the possibility of mislocalization in depth when the two locations at which elements are presented represent different positions in depth. Figure 1B displays the dot stimulus we employed. As shown, a square plane of dots was presented briefly twice at Position 1 (in depth), and then once at Position 2 (to the right of Position 1 and at a different position in depth). This stimulus is demonstrated in Movie Clip 2 (view through red–green filters. Looping the video provides a stronger demonstration of saltation). Note that in the experiments, this stimulus was embedded in a background of dynamic random dots to ensure that saltation is only evident through binocular fusion. Background dots were omitted to highlight the saltation elements for illustrative clarity. In Figure 1B, we outline the three most immediately obvious theoretically possible percepts arising from this stimulus configuration. First, because elements are presented at different depths, the visual system might use this difference in depth as a segmentation cue, resulting in no perceptual grouping and no saltation in depth (i). In terms of this first possibility, depth information is frequently used to segment an object from its background; a similar operation may act to disrupt grouping, breaking the visual saltation illusion. Second, visual saltation may occur such that the second element in the temporal sequence is perceptually mislocalized to the middle of space between the two locations but remains at its veridical depth position (ii). Under these circumstances, while perceptual grouping may result in a 2D mislocalization, depth is treated independently and is not affected by the mechanisms underlying visual saltation. Finally, 2D perceptual mislocalization of the second element might occur and also be accompanied by a mislocalization in depth (iii). This percept would be consistent with perceptual grouping producing a stimulus that resembles a single object traversing 3D space. Through measurement of the perceived position of elements in depth, the specific aim of the present study was to determine which of the above possibilities characterizes the perception of visual saltation in depth, and to examine the impact of the position in depth, and motion adaptation, on visual saltation in depth. 

Figure 1

 

(A) A schematic diagram of our stimulus is shown; three square dot planes are presented twice at position 1 and once at position 2. (B) Three possible percepts of (A). (i) Stimuli are seen at their veridical positions. (ii) Two-dimensional saltation is observed, with the second element mislocalized midway between the two elements only in the 2D image plane but not in depth. (iii) Saltation in depth observed with simultaneous mislocalization in 2D space and in depth.

Experiment 1: Cyclopean visual saltation characterized by mislocalization in 2D as well as in depth

As mentioned, the 2D visual saltation illusion is a compelling example of motion perception, most likely arising from grouping and filling in. The purpose of Experiment 1 was to determine whether the perceptual mislocalization that characterizes saltation could be generated when the two veridical positions are separated in 3D space. We expected that, if perceptual grouping occurs such that the visual system treats the stimulus as a single object traversing depth, intermediate elements in the saltation sequence would appear mislocalized to intermediate locations in 2D space and in depth. To be thorough, we examined visual saltation in depth for different depth pedestals, that is, as a function of the position of the stimulus away from point of fixation in depth in crossed and uncrossed directions. It is well known that stereoacuity is greatest around the point of depth fixation, and that sensitivity decreases as a function of distance away from fixation (Badcock & Schor, 1985; Blakemore, 1970; Ogle, 1953; Ogle & Weil, 1958; Siderov & Harwerth, 1995). In line with the interpretation that visual saltation arises from the grouping that occurs when spatial form is ambiguous (as it is in the periphery), we expected that saltation in depth would be most compelling when the stimulus is presented away from the depth fixation point where stereoacuity is coarse. 

Methods

Observers

Six experienced observers (aged 21–34 years) participated in Experiment 1. All had normal or corrected-to-normal visual acuity with no history of visual disorders. Two (SKK and JP) were authors on the study, while the others (DL, NY, SK, and SH) were naive to the purpose of the study. 

Stimuli

The stimuli were red–green anaglyph dynamic random-dot stereograms depicting an orthographically presented square (6° × 6°, 1 pixel = 0.024 cm) defined by 120 circular anti-aliased dots (80 cd m⁻²; diameter 0.11°) that occupied randomly chosen, non-overlapping, positions on a gray background (30 cd m⁻²). To prevent the tracking of individual dots, all dots were generated asynchronously and had a limited lifetime of 0.05 s. When they expired, dots were replotted back into the square to a random position. In addition to occupying random 2D positions, dots also occupied random positions in depth around the point of fixation. This was facilitated by assigning a random horizontal disparity difference value of between −0.025° and 0.025° of visual angle individually to corresponding dots in the red and green images. The geometric depth produced by these disparities can be approximated by 

$\eta \approx I\delta /D^2,$

(1)

where η is the disparity, I is the interocular separation (6.3 cm; French, 1921), δ is the simulated depth from the point of fixation, and D is the viewing distance (60 cm). Accordingly, the depth range of the square plane occupied a depth space of ±0.25 cm around the depth fixation point and the dot density of the stimulus was 6.67 dots cm⁻³. This dynamic random-dot stimulus occupied the depth plane coinciding with the point of fixation and was the background upon which stimuli were presented to elicit visual saltation. Additionally to aid fusion and to control for vergence, the stimulus was presented within a black square outline (8° × 8°, width: 0.2°) that coincided with the fixation plane at 0 disparity. 

Visual saltation was produced as follows: within the dynamic random-dot stereogram, three square planes of dots (1 × 1°) were created by assigning dots within the corresponding region of the dynamic random-dot stereogram a different disparity value to those used to generate the dynamic background. These were presented briefly (for 0.3 s, making the duration of the stimulus sufficient for optimal stereoscopic perception, e.g., Harwerth, Fredenburg, & Smith, 2003; Tyler, 1991), two to the first location (2° to left), and one to the second location (2° to the right of the horizontal and vertical midpoints of the dynamic random-dot stereogram). Thus, the 2D separation of the two locations was 4° of visual angle. In addition to this 2D separation, the squares presented at these two locations were separated in depth by ensuring that dots corresponding to the first and last locations had different disparities. The relative disparity difference at the two locations was kept constant at 0.3°; approximated by Equation 1, this disparity value corresponded to a depth value of 3 cm. Depending on the condition, the first location was “closer” to the observer in depth and the second location was “further,” or vice versa. Accordingly, the direction of saltation was either toward or away from the observer (see Procedures section). These stimulus construction procedures ensured that the saltation was cyclopean; the monocular images have only randomly moving dots and must be combined for the stimulus to be apparent. This ensured that any perceptual mislocalization observed with saltation in depth is driven primarily by disparity-tuned mechanisms, and not derived solely from monocular cues. 

Each element of the saltation sequence was interleaved with a period in which only the random-dot background was seen. The duration of this period corresponded to the ISI, which was kept constant. Previous research (e.g., Khuu et al., 2010), as well as our pilot investigations using the above described stimuli, has shown that ISI is the most important parameter to the perception of this illusion, and that the illusion is most compelling for very brief ISIs of approximately 0.25 s (see Experiment 2). We therefore used an ISI of 0.25 s for this experiment. Stimuli were generated using MATLAB version 7 and displayed on a linearized 24-inch Mitsubishi Diamond Pro monitor driven at a frame rate of 120 Hz. Observers viewed the stimulus binocularly at a viewing distance of 60 cm through red and green neutral density filters that ensured that only one image was seen by each eye. 

Procedure

The stimulus was presented to the observer such that the central point of the random-dot stereogram was always located 6° to the right of the fixation point, which was in turn indicated by a black cross at the center of the screen. It is important to note that at this horizontal stimulus eccentricity, depth discrimination is certainly possible, though stereoacuity is comparatively coarser than central vision (e.g., Rawlings & Shipley, 1969; Siderov & Harwerth, 1995). In each trial, observers were shown two saltation sequences in quick succession (with an intersequence interval of 0.25 s, note that the duration of the stimulus is dependent on ISI). After the second sequence ended, the stimulus disappeared from the screen and observers used a mouse probe (a black spot with a radius of 0.15° of visual angle) to indicate the perceived 3D position of either the first, second, or last element of the sequence. To facilitate this judgment, an auditory “beep,” coinciding with the to-be-judged element, sounded. For each trial, two judgments were made. Observers were first required to indicate the 2D position of the cued element (coinciding with the beep) by placing the probe over the perceived 2D position, and then to judge its position in depth by adjusting the perceived position in depth of the probe (by pressing two buttons on a keyboard, which changed the binocular disparity of the probe in the red and green images). The probe was not visible on the screen during the stimulus presentation, rather it appeared at a random location on the screen at the offset of the second sequence and disappeared immediately after the observer had pressed the mouse button. Observers were requested to respond as quickly and as accurately as possible. Prior to the main experiment, observers were given many practice trials sufficient to familiarize themselves with the task, which minimized judgment errors. 

In separate conditions, we repeated the stimulus procedures for saltation stimuli presented at different depth pedestals relative to the depth fixation point. Stimuli were presented at different depth pedestals in both crossed (positive values) and uncrossed (negative values) directions with the disparity at the first location to −0.6, −0.3, −0.15, 0, 0, 0.15, 0.3, and 0.6° (producing inferred depth positions of −6, −3, −1.5, 0, 0, 1.5, 3, and 6 cm from the fixation plane). As mentioned, the second location was at a different depth plane and was placed 3 cm in depth in front of the first location for crossed disparities, or 3 cm behind it for uncrossed disparities. This depth difference was produced by ensuring that, in addition to the depth pedestals noted above, dots at the first and second locations differed in disparity by 0.3° for crossed and −0.3° for uncrossed disparities. Therefore, in both cases, saltation was always from left to right; however, for crossed disparities saltation approached the observer while for uncrossed disparities saltation receded in depth from the observer. A block comprised 240 trials: eight stimulus depth pedestals for both crossed and uncrossed directions, for each of three position judgments (first, second, and last), repeated 10 times. Stimulus conditions were randomized within and between each block. Observers each completed 5 blocks such that each condition had 50 trials. Results were averaged across the 50 trials for each condition. 

Results

The results for the six observers were very similar and were therefore averaged. These results are given in Figure 2, which plots the judged position in terms of Cartesian X and Z depth coordinates of the first, last, and second elements (given by the schematics above each data cluster), relative to the X and Z positions of the first location. Data are given separately for crossed (Figure 2A) and uncrossed directions (Figure 2B). Different gray symbols represent data for different depth pedestals. Our data did not reveal any systematic change in the Y position of the stimuli. Therefore, for simplicity, these data are not illustrated in the figure. Dashed lines indicate the “veridical” positions of the first and second locations to which elements were presented. 

Figure 2

 

The perceived positions of the first, second, and last elements of the saltation sequence (in terms of Cartesian X and Z and indicated by schematics above each data cluster; N.B. for the second element, an outline is given to indicate its physical position) for (top) crossed and (bottom) uncrossed disparity positions; error bars represent one standard error of the mean (SEM). Dashed lines indicate the physical depth positions of the first and second locations.

Figure 2 presents a number of findings. It is worth noting first that error bars were small (judgments varied only within approximately 0.5° for X position, and 1 cm (corresponding to a binocular disparity of 0.1 deg) for depth position) indicating that observers were accurate in judging the position of elements despite their location in the periphery, though the variance for judgments of position in depth was larger than in previous studies (see, e.g., Siderov & Harwerth, 1995). The latter outcome is most likely due to the nature of the stimulus and task used by the present study. Unlike Siderov and Harwerth (1995) who required to observers to judge the apparent depth difference between two sine-wave stimuli in a Vernier task, our study required observers to judge the depth position of a briefly presented object at the end of the stimulus presentation period. The more subjective nature of our task may lead to greater variability in responding since there is no “physical” stimulus for comparison. Regardless, observers were reliable in judging the position of the stimulus in depth. Additionally, it is important that the results for crossed and uncrossed positions were very similar to each other and were mirror opposites, reflecting the difference in direction of saltation in depth. 

Across all conditions, the perceived positions of the first and last elements corresponded to their veridical positions. Data for these two judgments were clustered around the corresponding dashed lines indicating the veridical positions of the first and last elements. In contrast, characteristic of the saltation percept, the perceived position of the second element was non-veridical for all conditions; it was laterally mislocalized in the 2D image plane (i.e., X position) to a position directly between the two veridical positions (despite the fact that its physical origin is at the first location) regardless of the depth pedestal (i.e., position in depth, disparity) of the saltation stimulus. A 2 × 4 repeated measures ANOVA was conducted comparing the perceived average 2D position of the first and second elements of the visual saltation sequence at the 4 different depth pedestals. This analysis confirmed the significant difference between the perceived positions of the first and second elements (crossed disparities: F(1,20) = 284.13, p < 0.001; uncrossed disparities: F(1,20) = 289.66, p < 0.001), but not position in depth (crossed disparities: F(3,20) = 0.52, p = 0.667; uncrossed disparities: F(3,20) = 1, p = 0.4025). Furthermore, the non-significant interaction between the two factors (crossed disparities: F(3,20) = 1.58, p = 0.2095; uncrossed disparities: F(3,20), p = 0.2516) indicates that the extent of the difference in perceived 2D positions of the first and second elements is the same across depth pedestals. Bonferroni posttests (Neter, Wasserman, & Kutner, 1990) between the perceived positions of the first and second elements revealed significant differences across all depth position conditions (p < 0.001). This result reiterates Khuu et al.'s (2010) finding that 2D saltation is compelling at an ISI of 0.25 s, but more importantly shows that placing elements in depth (in either crossed or uncrossed directions) does not disrupt the appearance of the illusion. The hypothesis given in Figure 1Bi is obviously inconsistent with these data (collected under the stimulus conditions outlined in Experiment 1), so we rule it out as a characterization of 3D saltation. 

Most interestingly, while the second element was laterally displaced in the 2D image plane, its perceived position in depth was dependent on the depth pedestal of the stimulus. For small disparities (resulting in depth pedestals near the fixation point), the perceived position in depth of the second element was near its actual depth position (horizontal dashed lines). However, as the depth pedestal increased, the second element was perceptually mislocalized away from its veridical depth position and was judged to be midway in depth between the two locations (approximately 1.5 cm for positions furthest away from the depth fixation point). As above, 2 × 4 repeated measures ANOVA was applied, this time to gage the impact of depth pedestal on the judged positions in depth of the first and second elements. There was a significant difference between the perceived positions of the first and second elements in depth (crossed disparities: F(1,20) = 6.54, p < 0.001; uncrossed disparities: F(1,20) = 4.32, p < 0.05), and for depth pedestal (crossed disparities: F(3,20) = 23.81, p < 0.001; uncrossed disparities: F(3,20) = 29.86, p < 0.001). Additionally, the interaction was significant (crossed disparities: F(3,20) = 4.33, p < 0.01; uncrossed disparities: F(3,20) = 2.87, p < 0.05), indicating that the positional difference between the judged positions in depth of the first and second elements is not the same across all disparities. Bonferroni posttests confirmed that the differences were significant only at the larger crossed depth pedestals of 0.3° (t(25) = 3.632, p < 0.01) and 0.6° (t(25) = 3.889, p < 0.01) and uncrossed depth pedestals of −0.3° (t(25) = 4.305, p < 0.001) and −0.6° (t(25) = 4.056, p < 0.001). Thus, for these comparatively larger depth pedestals, the second element was perceptually mislocalized in the image plane, as well as in depth. This percept would be analogous to that of a single object traversing 3D space in between the two locations. As mentioned, this outcome is most likely because stereoacuity decreases as a function of distance in depth away from fixation. The visual system is able to accurately resolve the depth order of stimuli at points near fixation; therefore, perceptual grouping leading to saltation does not manifest as mislocalization in depth. However, while presentation of the stimulus 6° to the right of fixation results in an immediate reduction in stereoacuity, as the stimulus is systematically moved away from the point of fixation, stereoacuity decreases, reducing the ability of the visual system to accurately resolve depth information. To resolve resulting ambiguities, it interprets the stimuli as a single stimulus that traverses depth between the two locations. It is important to note that given that stereoacuity and 2D spatial acuity mutually decrease as a function of retinal eccentricity it is likely that, at much greater eccentricities, the percept of saltation in depth will be observed for small depth pedestals. 

Together, these data show that visual saltation in depth does not automatically accompany saltation in the 2D image plane, suggesting separate processes. As shown in Figure 2, 2D mislocalization was reported across all depth positions, but mislocalization in depth was evident only for positions in depth away from fixation. This suggests that the visual system is able to independently segregate the two percepts. These findings reveal a complex grouping procedure that integrates different visual attributes but is dependent on the reliability or clarity of the visual attribute. Therefore, in relation to Figure 1B, these data confirm that the hypothetical percepts depicted in Figures 1Bii and 1Biii are both possible—which of these is observed is dependent on stereoacuity and the position of the stimulus from fixation. 

Experiment 2: Visual saltation exclusively in the depth plane

Experiment 1 showed that, when saltation occurs at crossed and uncrossed disparities away from fixation, the second element is mislocalized in depth to a position directly between the two physical depth locations. However, the stimulus configuration in that experiment had the two locations of stimulus presentation offset laterally in the image plane as well as in depth. Because stimuli were laterally displaced, it is possible that the saltation in depth arose first from (and is contingent on) 2D grouping followed by an inference of depth position. Accordingly, it remains to be seen whether it is possible to obtain saltation with stimuli solely on the basis of binocular disparity differences. Experiment 2 investigated this issue by repeating the procedures of Experiment 1 with stimuli for which there was no lateral displacement, only separation in depth. 

Procedures

Three observers participated in Experiment 2. One (SKK) was the author, while NY and SH were experienced observers naive to the purpose of the experiment. As in Experiment 1, observers judged the position in depth of the first, second, and last elements of the sequence for different depth pedestals of −0.6, −0.3, −0.15, 0, 0, 0.15, 0.3, and 0.6°, which corresponded to the position of the first location, and these disparity values corresponded to depth positions of −6, −3, −1.5, 0, 0, 1.5, 3, and 6 cm from the fixation plane. The second location was offset in depth by 3 cm and was “behind” the first site of stimulation for uncrossed disparities (producing “receding” saltation) or in front (producing “approaching” saltation) for crossed disparities. In a block of trials, observers judged the positions of the three elements of the saltation sequence at eight depth positions of the stimulus, 10 times. Thus, there were 240 trials per block, with conditions randomized both within and between blocks. Observers each completed five blocks. 

Results

The results of Experiment 2 are shown in Figure 3, which plots the perceived depth position of the first, second, and last elements (different gray levels) relative to the position of the first element, as a function of stimulus depth pedestal (X-axis, given as binocular disparity in degrees). Results for the three observers are shown separately (different symbols); dashed lines mark averaged data. 

Figure 3

 

The perceived position in depth for the first, second, and last elements (different gray levels) is plotted as a function of the depth pedestal of the saltation stimulus (values correspond to the position of the first location). Different observers are given by different symbols, while differently shaded dashed lines denote averaged data. Error bars indicate one SEM. The results for (A) crossed and (B) uncrossed directions are given separately.

Figure 3 shows that the pattern of results is similar for all observers and mirrors that of Experiment 1. First, the perceived positions of the first and last elements of the sequence did not change as a function of the depth pedestal in both crossed and uncrossed directions (Figures 3A and 3B). Judged positions for the first and last elements corresponded well to their actual depth positions given by the dotted lines. However, when the stimuli were near in depth to the point of fixation (e.g., 0°), the second element was perceived at or near its physical position, indicating no saltation. However, as the stimulus depth pedestal increased, there was a systematic mislocalization in depth of the second element between the two locations. This mislocalization was such that, at depth pedestals of −0.3° and 0.3°, the second element was judged to be approximately midway in depth. Repeated measures ANOVA confirmed the significant difference between the perceived positions of the first and second elements (crossed disparities: F(1,8) = 109.35, p < 0.001, uncrossed disparities: F(1,8) = 271.73, p < 0.01) and depth pedestal (crossed disparities: F(3,8) = 26.90, p < 0.001, uncrossed disparities: F(3,8) = 59.18, p < 0.01). The interaction was significant (crossed disparities: F(3,8) = 16.75, p < 0.001, uncrossed disparities: F(3,8) = 46.2, p < 0.01): Bonferroni posttests revealed significant differences for both crossed and uncrossed disparities at absolute depth positions of 0.15° (crossed disparities: t(10) = 6.164, p < 0.01, uncrossed disparities: t(10) = 3.703, p < 0.05), 0.3° (crossed disparities: t(10) = 7.256, p < 0.01; uncrossed disparities: t(10) = 14.18, p < 0.01), and 0.6° (crossed disparities: t(10) = 9.652, p < 0.01; uncrossed disparities: t(10) = 13.08, p < 0.01). These findings confirm not only that it is possible to produce visual saltation in the Z-depth plane, but also that the percept, while comparable to that noted in Experiment 1, is not dependent on 2D mislocalization. These results mirror previous reports of apparent motion exclusively along the Z-axis (Norcia & Tyler, 1984). 

In Experiment 2, stimuli were presented at a fixed retinal eccentricity of 6 degrees and it was found that, for depth positions sufficiently far from fixation, the second element is mislocalized in depth midway between the first and last. While we have noted that the appearance of the 2D visual saltation illusion is dependent on retinal location eccentricity (see Geldard, 1982; Moradi & Shimojo, 2004), it is not known whether and how visual saltation in depth is dependent on retinal eccentricity. It is known from the work of Badcock and Schor (1985) and Siderov and Harwerth (1995) that stereoacuity decreases with retinal eccentricity. We therefore predict that saltation in depth would also be dependent on retinal eccentricity such that the illusion is evident only at sufficiently large eccentricities. Accordingly, we conducted a supplementary experiment that repeated Experiment 2, with stimuli presented at one depth pedestal (a crossed direction of 0.6°), but at retinal eccentricities of 0, 3, 6, and 9°. We chose this depth pedestal because it produced compelling saltation in depth in Experiments 1 and 2. Following the methods of Experiment 2, observers were required to judge the perceived positions of the first, second, and last elements of the sequence. The same observers as in Experiment 2 and an additional experienced observer (JE) participated in this supplementary experiment. 

The results of this supplementary experiment are shown in Figure 4. The perceived depth positions of the first, second, and last elements of the saltation sequence are plotted as a function of retinal eccentricity. The pattern of results is similar for all four observers. The judged position of the first and last elements of the sequence does not change with retinal eccentricity, though more variability in the data is apparent for larger eccentricities. Clearly though, Figure 4 shows that the perceived position of the second element is dependent on retinal eccentricity. As retinal eccentricity is increased, the second element is perceptually mislocalized, and for retinal eccentricities of 6 and 9°, it is perceived at a position midway between the first and last elements. A two-way repeated measures ANOVA was performed to gage the significance of the differences between the mean positions of the first and second elements for different retinal eccentricities. We note a significant difference between the perceived positions of the first and second elements (F(1,6) = 132.5, p < 0.001) and for retinal eccentricity (F(3,18) = 19.99, p < 0.001). Additionally, a significant interaction effect was noted (F(3,18) = 21.35, p < 0.001) suggesting that the extent of difference of the position in depth of the first and second elements was not the same across different retinal eccentricities. Bonferroni posttests reported significant differences for 6° (t(24) = 1.485, p < 0.001) and 9° (t(24) = 1.444, p < 0.001) of eccentricity. Like 2D saltation, then, saltation in depth is dependent on retinal eccentricity, and the illusion occurs when there is ambiguity brought about through peripheral presentation where stereoacuity is coarse. 

Figure 4

 

The perceived position in depth of the first, second, and last elements (different gray symbol levels) of the visual saltation sequence plotted as a function of retinal eccentricity. Points represent data for individual observers. Dashed lines indicate the perceived position of the two physical sites of stimulation. Error bars denote one SEM of the averaged data.

Experiment 3: The effect of adaptation on the perceived position of mislocalized elements of the saltation in depth illusion

In the previous experiments, we showed that it is possible to generate a saltation illusion that traverses depth. As previously noted, Shore et al. (1998) suggested that, via the process of perceptual grouping, the stimuli are together resolved as a single object that travels across the space between the two locations. This interpretation suggests the operation of high-level mechanisms; the interpretation of motion in depth is produced at an advanced stage of processing and is then fed back to lower cortical areas that actively code the retinotopy to produce the percept. Neural mechanisms between the two locations are not, therefore, activated by physical stimulation but by inputs (through feedback connections) from higher cortical areas. If this account does indeed explain the perception of visual saltation, it would be expected that the introduction of bias to the response of local mechanisms coding depth in the space between the two locations would lead to a distortion in the perceived position of the mislocalized element. In a recent study, Khuu et al. (2010) confirmed that this effect occurs with 2D saltation. Using motion adaptation, Khuu et al. showed that adaptation at the perceived position of the mislocalized element results in displacement of its perceived position in the MAE direction, but that adaptation at its physical position produces no distortion. Khuu et al.'s results therefore confirmed that the percept of saltation is analogous to that of a real object physically traversing space between the two locations, consistent with Shore et al.'s (1998) suggestion. In Experiment 3, we investigated this issue in relation to saltation in depth, specifically asking whether motion adaptation in depth distorts the perceived position of elements that are mislocalized in depth. Previous research has shown that it is possible to generate an MAE with stereoscopic motion in depth (e.g., Patterson, 1999; Patterson, Bowd, Phinney, Fox, & Lehmkuhle, 1996) and we expected that, if the visual system interprets the elements as arising from a single stimulus that traverses depth, the perceived location of the mislocalized second element would be shifted in the MAE direction. 

Methods

Observers

Four observers participated in Experiment 3. One (SKK) was the author, while NY, SH, and LS were experienced observers who were naive to the aims of the experiment. 

Stimuli

Stimuli were presented in a similar configuration to Experiment 1. That is, the first and last elements were displaced both laterally and in depth and were presented at a crossed disparity of 0.6° (6 cm in depth) and at an ISI of 0.25 s for which compelling saltation in depth was noticed in the previous experiments. We used this configuration, rather than that of Experiment 2, because previous research has shown that the MAE in depth is not dependent on the absolute disparity difference between elements, but on their subjective depth order (Sohn & Lee, 2009). It was therefore possible that an MAE generated in between stimuli separated only in depth (as in Experiment 2) would prevent individuation of the effect of motion adaptation on the first, second, and last positions of elements, leaving all elements distorted by the MAE since they all overlap with the adapted region. The stimulus configuration of Experiment 1 avoids this problem because there is no physical overlap between the two physical locations in either 2D or in depth with the adapting stimulus, making an element mislocalized in depth easily individuated. 

Procedure

Prior to the presentation of the saltation sequence, observers adapted to motion in depth presented at the midpoint between the two locations for 120 s. The adapting motion in depth stimulus was a cuboid (3D rectangle, X = 2°, Y = 2° Z = 3 cm (disparity range of 0–0.3°)) defined by randomly located dots (see Figure 4). Changing their binocular disparity generated motion in depth. The far end of the cuboid was positioned so that it was aligned in depth with the first location, while the near end was aligned with the second. These procedures of generating motion in depth result in an interocular velocity difference cue between the two stereo images. However, in each monocular image, dots are moving in a given direction, and over the adapting period, a local 2D MAE is generated. It is unlikely that this localized 2D MAE will influence the perceived position of the saltation sequence because the two locations at which elements are physically presented (to generate the percept of saltation in depth) are laterally separated and do not physically overlap with the effected region. 

On each trial, after adaptation, a probe (a Gaussian spot, similar to that used in the previous experiments) was presented to positions directly above (Y = 0.25°), and vertically aligned with, the perceived 2D position of the first, second, and last elements of the sequence (as judged in Experiment 1). At no point did the probe overlap with the adapted region, ruling out any possibility that a perceived position shift in depth is due to distortion of the probe position by the MAE. Additionally, observers used the probe to indicate the perceived position in depth of the first, second, and last elements of the saltation sequence. They did this by aligning the position in depth of the probe with the relevant element of the sequence and by adjusting its binocular disparity (at steps of 0.02° (corresponding to a simulated depth difference of 0.2 cm)). Each trial was ended only when the observer was satisfied that the probe was aligned in depth. As in Experiments 1 and 2, the probe was not visible on the screen during the stimulus presentation. Rather, it appeared at the offset of each sequence (to a randomly selected depth position (0–3 cm)) and disappeared immediately after the observer had judged it aligned. After each sequence presentation, motion adaptation “top-ups” of 5 s were given. 

In separate conditions, we varied the direction of the adapting stimulus so that dots moved either toward (approaching motion) or away from (receding motion) the observer (see Figure 5). Previous research has shown that the extent of position shift is dependent on the speed of the dots of the adapting stimulus (e.g., De Valois & De Valois, 1991; Khuu et al., 2010). Therefore, we examined the effect of dot speed, repeating the procedure for speeds in depth of 0.75, 1.5, 3, and 6 cm s⁻¹. We hypothesized that the extent of mislocalization in depth induced by the MAE would not only be dependent on the direction of motion but would increase with the adapting speed. Speed in depth, Vz, was approximated from a modification of Equation 1, 

$Vz=D^{2}V/I,$

(2)

where D is the viewing distance, I is the interocular separation, and V is the rate of change of disparity, which, to produce the aforementioned speeds, was 0.075, 0.15, 0.3, and 0.6° s⁻¹, respectively. A block of trials comprised 240 trials: two directions of motion in depth at four speed levels, for separate judgments of the first, second, and last elements of the sequence, repeated 10 times each. Stimulus conditions were randomized within and between each block. Observers each completed 5 blocks such that each condition had 50 trials. Results were averaged across the 50 trials for each condition. 

Figure 5

 

Potential outcomes for the perception of saltation in depth from localized motion adaptation to receding (Figure 4A) and approaching motions (Figure 4B). In each case, the location of the gray square represents the perceived position of the second element when saltation is generated in 3D, while the changed perceived position of that element by the MAE is given by the dotted white square.

Results

The results of Experiment 3 are shown in Figure 6, which plots the perceived depth position of the first, second, and last elements of the saltation sequence, as a function of the speed of the adapting stimulus. Figure 6A depicts the results for motion toward the observer, while Figure 6B illustrates the results for motion away. The results for the four observers are shown with different symbols while averaged data are given by differently shaded dashed lines. The pattern of results was similar for all four observers. The perceived position of the first and last elements of the saltation sequence was unaffected by motion adaptation regardless of the adapting speed and the direction of motion, and their judged positions coincided with their physical positions (depicted by the dotted lines). Because motion adaptation occurred between the two locations, these elements did not physically overlap with this region, and their perceived position was therefore unaffected. Despite larger receptive fields in the periphery, these results suggest that motion adaptation was confined to the midpoint in between the first and last elements. These findings are consistent with recent observations by McGraw and Roach (2008), who demonstrated that the MAE in the periphery is largely confined to the adapted area. Our data confirm this observation for motion adaptation in depth. 

Figure 6

 

The perceived position of the first, second, and last elements (different gray levels) is plotted as a function of the speed of the adapting stimulus for (A) receding motion and (B) approaching motion. Error bars represent one SEM.

The key observation to be drawn from Figure 6 is that the second element, unlike the first and last elements, was affected by motion adaptation—a pattern of results that conforms to the predictions given in Figure 5. While the second element's physical location did not overlap with the adapted region, its perceived location did; under the conditions eliciting saltation in depth, the second element was perceived displaced to a position midway between the other elements, and was therefore perceived as overlapping with the motion adapted region, and was subject to the MAE. Importantly though, the extent of displacement of the second element caused by the MAE was dependent on the adapting speed. Consider first Figure 6A, which shows the results for receding motion conditions. As the adapting speed increased, the perceived position of the second element was displaced in depth in the MAE direction. A one-way analysis of variance confirmed a significant effect of perceived position of the second element as a function of adapting speed (F(3,9) = 74.71, p < 0.001). When the adapting speed was slow (0.75 and 1.5 cm s⁻¹), the perceived position of the second element was approximately midway between the first and last elements, so while there was saltation there was no effect of adaptation. However, as adapting speed increased, the second element appeared displaced in depth in the MAE direction and toward the judged position of the third element and the observer. Bonferroni posttests between the mean for the slowest speed (0.75° s⁻¹) and faster speeds revealed significant differences for speeds of 0.3 (t(12) = 8.599, p < 0.001) and 0.6° s⁻¹ (t(12) = 13.67, p < 0.001), and visual inspection of these data would indicate it is perceived shifted in the MAE direction. The pattern of results is complementary for approaching motion, as shown in Figure 6B. When the adapting speed is slow, there is no position shift consistent with an MAE; the second element is perceptually mislocalized to the location between the first and last elements. However, for fast speeds, its perceived position is displaced in the MAE direction (away from the observer and to the position of the first element). This effect is significant, F(3,9) = 44.36, p < 0.001, with significant differences in the mean of the slowest speed of 0.75° s⁻¹ and faster speeds of 0.3 (t(12) = 6.858, p < 0.001) and 0.6° s⁻¹ (t(12) = 10.47, p < 0.001). 

It is important to note that motion adaptation does not destroy the perception of saltation in depth. We point this out with relation to Figure 6B (approaching motion) because close inspection might at first suggest that the illusion is broken down by the MAE, returning the perceived location of the misplaced second element back to its physical location as adapting speed increases. However, the distortion in depth position due to the MAE is dependent on the direction of adapting motion. Figure 6A shows that adapting to receding motion displaces the position of the second element in depth away from its physical location at the first location with the adapting speed; if increasing the speed of the MAE destroyed saltation, results for this condition would be identical to those in Figure 6B. 

We showed in Experiment 3 that the second element is subject to the MAE in depth at the brief ISI of 0.25 s and noted that this is likely because the perceived location of the mislocalized element (rather than the physical location) overlaps with the adapted area. Confirmation of this interpretation can be achieved by considering whether the second element is subject to an MAE in depth when elements are presented at a much longer ISI, one that does not generate the saltation illusion. At long ISIs, the second element ought to be localized veridically, and therefore motion adaptation in the region between the first and last elements ought not to displace the second element in the MAE direction because there would be no overlap between the perceived location of elements and the adapting area. We tested this prediction in a supplementary experiment that repeated the procedures of Experiment 3 (only for the approaching motion condition) with an adapting speed of 6 cm s⁻¹, but with elements presented at an ISI of 0.75 s. Our pilot study demonstrated that this ISI does not generate saltation; elements are perceived at their veridical 2D and depth positions. In this experiment, observers were required to judge the perceived position in depth as well as in 2D space. To prevent observers from placing the probe in the motion adapted region (making it subject to the MAE), it was placed 0.25° above the saltation sequence. Observers used keyboard button presses to horizontally slide the position of the probe to the left or to the right until it appeared aligned with the to-be-judged element. As in Experiment 3, the position in depth of the probe could be adjusted by changing its disparity, again via a keyboard button press. On each trial, the probe initially appeared at a randomly selected X position and Z depth position (X: 0–5°; Z: 0–3 cm) and disappeared when the observer deemed the probe aligned with the to-be-judged element. Three observers completed this experiment: one (SKK) was the author, while the other two (SH and HS) were experienced observers naive to the purposes of the study. 

The results of this supplementary experiment are shown in Figure 7, which plots perceived positions of the first, second, and last elements of the saltation sequence. Results from all observers conformed to the same simple pattern: the first, second, and last elements were perceived at their physical positions. The second element was obviously not affected by motion adaptation because its perceived position coincides with its veridical position, which does not perceptually overlap with the motion-adapted region. These findings contrast markedly with those given in Figure 6 and demonstrate that mislocalization in depth arising from motion adaptation relies on the perception of a stimulus within the motion-adapted area. 

Figure 7

 

Saltation presented at an ISI of 0.5 s. The perceived position of the first, second, and last elements (different gray levels) of the saltation sequence is plotted as a function of their X position (SKK—squares; SH—triangle; HS—circle). Error bars represent one SEM.

In summary, Experiment 3 showed that, under conditions sufficient to generate saltation in depth, the second element was perceived displaced in the MAE direction if the adapting speed was faster than approximately 1.5 cm s⁻¹. However, when saltation was not generated due to a lengthy presentation, displacement of the second element due to the MAE did not occur. The key difference between these two conditions is the perceived location of the second element: where saltation is generated and the second element is perceived as localized directly between the other two elements, there is overlap with the adapted region and displacement in the MAE direction occurs. Where a failure to generate saltation results in physical localization of the second element, there is no overlap with the adapted region and therefore no displacement due to MAE. 

General discussion

The purpose of this study was to investigate whether the visual saltation illusion occurs across 3D space and to examine the perceptual mechanism that might underlie it. More specifically, we sought to examine whether the 3D visual saltation illusion can be explained in the same way as 2D saltation—by the perceptual grouping of elements whose spatial location is made ambiguous by peripheral and rapid presentation, and subsequent interpretation of the sequence as arising from a single object moving smoothly across the space between the two locations (e.g., Khuu et al., 2010; Shore et al., 1998). There were a number of clear findings. First, in Experiments 1 and 2, we demonstrated that it is possible to generate a percept of visual saltation in depth; the mislocalization of intermediate elements of the saltation sequence to intermediate depth positions was compelling. Importantly though, this percept was dependent on the position of the stimulus in depth, with mislocalization only evident when the stimulus was presented away from the point of depth fixation—where coarse stereoacuity ensures ambiguity in the position of elements in depth. In Experiment 3, we demonstrated that adaptation to motion in depth at the location between the first and last elements results in distortion in the MAE direction only for the second element when the ISI is sufficient to produce saltation, but not when the ISI is too long to facilitate saltation. This finding fits well with the hypothesis that perceptual grouping underlies saltation as the location of motion adaptation coincides with the perceived location of the second element during saltation, and results were akin to those that would be expected if the elements arose from a moving stimulus. 

As noted, previous studies examining perceptual grouping have reported the phenomenon of apparent motion from briefly presented objects presented in depth (Atteneave & Block, 1973; Julesz & Bosche, 1966; Norcia & Tyler, 1984; Regan & Beverley, 1973; Tse & Logothetis, 2002). For example, using random-dot stereograms to generate a cyclopean stimulus, Norcia and Tyler (1984) reported compelling apparent motion between two locations defined only by a difference in disparity. Additionally, Atteneave and Block (1973) showed that a depth perspective background produces apparent motion in depth. These observations suggest that perceptual grouping is reliant on a variety of depth cues besides binocular disparity to derive motion in depth. Our finding that saltation can also be produced in depth is in strong agreement with this past research on apparent motion. Together these findings reveal that the visual system is very capable of deriving a percept of image motion through the grouping of objects distributed in 3D space. Note though that these findings differ in that saltation is only evident for peripheral presentations, whereas apparent motion is perceptible in central vision. 

The generation of saltation in depth, as documented in the present study, raises the important issue of the processes by which the percept arises. It is unlikely that the illusory percept can be accounted for by the operations of low-level spatiotemporal mechanisms for a number of reasons. First, the stimulus used in the present study is more likely to activate “long-range” motion processes that reflect the operation of high-level grouping and the feature tracking of objects over a large spatial distance (Cavanagh & Mather, 1989). Second and more importantly, spatiotemporal filters will only signal directional motion between the second and last elements, and not between the first and second, since at this transition point, there is both spatial and temporal changes (Moradi & Shimojo, 2004). Consequently, it is not clear how low-level mechanisms can account for the positional mislocalization characterizing saltation or as noted by the present study, a change in position in depth. Third, as mentioned in the Introduction section the visual saltation illusion can be produced even under dichoptic presentations, and across the blind spot where no receptors exist to code information (Lockhead et al., 1980). Lastly, when elements at the first and last locations are different colors, visual saltation is accompanied by transformational changes, such as a color, which indicates that the percept is a product of “interpretation” and not derived or can be accounted for by low-level “bottom-up” processes (Geldard, 1982). Consequently, given these distinctions it is unlikely that low-level mechanisms can account for the “saltatory” motion percept that characterizes this illusion. 

While we have assumed grouping to be responsible for the percept of saltation, an important issue is what is the mechanism by which elements are grouped? A possible explanation is that the visual saltation illusion arises from visual attention and attentive tracking (e.g., Cavanagh, 1992; Horowitz & Treisman, 1994; Lu & Sperling, 1995; Wertheimer, 1912). It has been shown that attention modulates perceptual grouping (e.g., see Han, Jiang, Mao, Humpreys, & Gu, 2005), raising the distinct possibility that objects briefly presented at separate spatial locations are “grouped” as attention is shifted from one location to another (see, e.g., Dick, Ullman, & Sagi, 1987; Horowitz & Treisman, 1994; Shim & Cavanagh, 2004; Shioiri, Yamamoto, Kageyama, & Yaguchi, 2002). Such a process is thought to involve “top-down” projections in which higher cortical areas feed back to lower cortical areas to control and modulate grouping (Han et al., 2005; Treisman & Gormican, 1988). This explanation can be applied to account for the visual saltation illusion in the following way: because the stimulus is sufficiently in the periphery (and for fast ISIs), the spatial location of the second element is rendered ambiguous; however, attentional tracking would signal directional motion (as the spotlight of attention is moved from one location to the next), which acts to group the elements. Accordingly, the visual system makes the retrospective assumption that the second element arose between the two physical locations and the percept resembles an object “saltating” from one location to the other. This would be in agreement with our results. However (as shown in Figure 7), for long ISIs, and retinal eccentricities close to fixation, attentional grouping does not occur since the spatial locations of the elements are clearly evident; they are seen at their physical locations. In many ways, this explanation for visual saltation is in agreement with previous accounts of how visual attention can induce shifts in the perceived positions of briefly presented objects (see, e.g., Shim & Cavanagh, 2004, 2006; Whitney, 2002; Yamada, Kawabe, & Miura, 2008). However, a noteworthy point is that while it is likely that attention may play a role in grouping for both apparent motion and saltation, this process of grouping can lead to separate and distinct percepts of high-level motion. Finally, where the present study has contributed to understanding is that we have demonstrated that such a process exists for stimuli presented in depth. 

It is possible that attentional grouping is additionally facilitated by visual persistence: the tendency for a briefly flashed object to persist beyond the physical point of offset (Di Lollo & Bischof, 1995). Visual persistence would mean that elements of the visual saltation illusion appear closer in temporal proximity (since elements persist for longer, but each element is distinct), thus facilitating the grouping of elements. This would agree with the fact that the illusion is highly dependent on ISI, and for long ISI saltation in which visual persistence is not applicable, the illusion does not occur. 

The findings of the present study are in agreement with previous observations that higher order motion can result in the displacement of the perceived position of objects (e.g., Shim & Cavanagh, 2004; Watanabe, Nijhawan, & Shimojo, 2002; Watanabe, Sato, & Shimojo, 2003; Whitney, 2002). For example, Shim and Cavanagh (2004, 2006) have reported that the perceived position of a briefly flashed object at the center of an apparent motion sequence appears shifted in the direction of motion. Our findings are in line with their observations, showing that the representation of high-level motion can influence the spatial registry of objects, and that visual attention may feature in the perception of both illusions. However, while our study and those of Shim and Cavanagh report a mislocalization of the perceived position of elements, the percepts reported by Shim and Cavanagh and noted in the present study are perceptually different, reiterating our earlier point that while grouping in both illusions are mediated by attention, the interpretation of the each percept is different. For our stimulus, the visual system makes the interpretation of saltation with the second element assumed to be apart of the motion sequence, and subsequently, it is mislocalized between the two physical locations, while for Shim and Cavanagh, no saltation is observed, rather the second element appears to be separate from apparent motion, but its perceived position is shifted in the apparent motion direction. Consequently, the extent of the mislocalization in the two studies is largely different, with mislocalization noted in the present study much larger (approximately 2°) than those reported by Shim and Cavanagh (approximately 0.5°). Thus, while both studies report analogous position mislocalization, the differing percepts suggest separate high-level interpretation. 

It is proposed that the perception of the visual saltation illusion most likely arises from higher level processes, but no previous studies have sought to directly identify the exact cortical origins of this illusory percept, though previous electrophysiological investigations have noted that the saltation illusion elicits electrical change in parietal and frontal lobes (see Stogbauer, Wassenhove, & Shimojo, 2007). Given that the illusion involves the processing of spatiotemporal stimuli, its likely cortical origin is area Middle Temporal (MT), which has also been implicated in the perception of apparent motion. Neural imaging studies examining apparent motion have confirmed a functional arrangement in which the percept is generated in area MT (e.g., Liu, Slotnick, & Yantis, 2004; Muckli, Kohler, Kriegeskorte, & Singer, 2005; Pascual-Leone & Walsh, 2001) with activation then fed back to lower cortical areas such as V1 in which neural receptors corresponding to the area between the first and second locations on the retinotopy receive activation to produce the percept. Equivocal evidence for feedback projections from MT to V1 is available from neurophysiological investigations (e.g., Sillito, Cudeiro, & Jones, 2006). Furthermore, corroborating evidence exists showing that attentive tracking involves MT (Culham et al., 1998) and cortical feedback may be a means in which visual attention modulates cortical activation (see Anton-Erxleben, Stephan, & Treue, 2009; Treue & Maunsell, 1999), and thereby implicating the importance of attention in spatiotemporal grouping in line with the above explanation for visual saltation. While apparent motion and the saltation illusion are phenomenologically different (as discussed), they are similar in that they arise from the visual system interpreting a sequence of stimuli degraded by temporal frequency as representing a single object undergoing motion. In the case of the present study, higher cortical feedback would produce the percept of a single object traversing 3D space through simultaneously activation of neurons responsible for the coding of both position and binocular disparity. Psychophysical evidence for such channels has been provided by Regan and Beverley (1973; see Patterson, 1999 for a discussion), and it is well documented that neurons in primary visual area are sensitive to both binocular disparity and motion (e.g., Ohzawa, DeAngelis, & Freeman, 1996; Poggio & Fischer, 1977). Additional evidence implicating MT as a possible site for the high-level motion and depth processing has been provided by DeAngelis, Cummings, and Newsome (1998), who reported that electrical stimulation of disparity-tuned cells in area MT results in changes in the perceived depth position of random-dot stimuli in a direction consistent with the cell's disparity preference. The results of DeAngelis et al. demonstrate the causal role of MT cells in the explicit perception of depth. 

The results of Experiment 3 are consistent with this conceptualization of the neural underpinnings of saltation: adaptation between the two locations changes the responses of both disparity-tuned and position-coding neurons at that location, which in turn disrupts the neural representation of the illusion, which leads to a distortion in the saltatory path in depth. Further evidence for an MT locus in the coding of high-level motion and position (and therefore saltation) is provided by McGraw, Walsh, and Barrett (2004), who reported that transcranial magnetic stimulation (TMS) to area MT disrupts the effect of motion adaptation on perceived position, but that stimulation to V1 does not. In light of this, it would be worthwhile to examine the effect of TMS on the perception of saltation in depth, given the added finding that depth and motion are mutually processed at MT (e.g., Ponce, Lomber, & Born, 2008). We would predict in this case that TMS disruption to MT will break down visual saltation in depth, but this is speculative. 

Two-dimensional shifts in the perceived position of an object in the MAE direction after a period of 2D motion adaptation have been documented since Snowden (1998) and Nishida and Johnston (1999; see Whitney, 2002, for a review). However, the present study represents the first attempt that we are aware of at generating the MAE effect in 3D. As Experiment 3 showed, adaptation to motion in depth leads to localized distortion of the perceived position of an element in the MAE direction in depth. Despite the novelty of this experimental procedure, it fits well with an established literature that hints at the interaction between “real” motion and perceived position in depth. For example, Tsui, Khuu, and Hayes (2007) showed that the perceived position of a 3D cylinder containing dots moving in depth is shifted in the direction analogous to the same effect in the 2D image plane (e.g., De Valois & De Valois, 1991). Similarly, Edwards and Badcock (2003) noted that 2D radial motion presented at a particular depth plane shifts the perceived position of a stimulus in depth consistent with the direction implied by the complex motion. Together with the present results, these studies indicate that the visual system is well attuned to deriving position in depth and image motion and suggest that these processes represent separate subsystems (see Patterson, 1999), which are clearly interactive in the representation of 3D space. Additional evidence for such an interaction comes from the study by Lee, Khuu, Li, and Hayes (2008), who generated a compelling flash–lag effect with receding and approaching motions defined by changing disparity such that the flashed object appeared to lag behind the moving stimulus. An approaching stimulus resulted in a flashed stimulus appearing further away from the observer, while with receding motion, the flashed object lagging behind the object appeared nearer in depth. The present study joins with Lee et al. (2008) in revealing that the computation of depth position is influenced by image motion. 

In summary, the present study documents the generation of a compelling 3D version of the visual saltation illusion using a cyclopean stimulus. Our collective results comply with the hypothesis that saltation is the product of perceptual grouping of spatially ambiguous elements and the subsequent interpretation that the elements arose from a single stimulus moving smoothly across the space between the first and last stimulus locations. The distortion of the position of the second element in the MAE direction we observed in Experiment 3 is exactly what would be predicted from this hypothesis. These findings are consistent with a “top-down” process in which saltatory motion is generated in lower cortical areas through feedback projections from higher cortical areas, possibly involving attentive tracking and cortical area MT. 

Supplementary Materials

Supplementary Movie - Supplementary Movie 

Supplementary Movie - Supplementary Movie 

Acknowledgments

We thank the observers who participated in the study. This research was supported by an Australian Research Council (ARC) Discovery Project Grant (Grant Number DP110104713), an Early Career Researcher Grant, and a Science Faculty Research Grant from the University of New South Wales to S. Khuu. J. Phu was supported by a Faculty of Science Summer Research Scholarship from the University of New South Wales. We thank the two anonymous reviewers for their helpful suggestions and comments. 

Commercial relationships: none. 

Corresponding author: Sieu K. Khuu. 

Email: [e-mail]. 

Address: The School of Optometry and Vision Science, The University of New South Wales, Sydney, New South Wales 2052, Australia.