Feature integration within and across visual streams occurs at different visual processing stages

Abstract

Abstract

Abstract:

Abstract  Direction repulsion is a perceptual illusion in which the directions of two superimposed surfaces are repulsed away from the real directions of motion. The repulsion is reduced when the surfaces differ in dorsal stream features such as speed. We have previously shown that segmenting the surfaces by color, a ventral stream feature, did not affect repulsion but instead reduced the time needed to process both surfaces. The current study investigated whether segmenting two superimposed surfaces by a feature coprocessed with direction in the dorsal stream (i.e., speed) would also reduce processing time. We found that increasing the speed of one or both surfaces reduced direction repulsion. Since color segmentation does not affect direction repulsion, these results suggest that motion processing integrates speed and direction prior to forming an object representation that includes ventral stream features such as color. Like our previous results for differences in color, differences in speed also decreased processing time. Therefore, the reduction in processing time derives from a later processing stage where both ventral and dorsal features bound into the object representations can reduce the time needed for decision making when those features differentiate the superimposed surfaces from each other.

Introduction

An object in the visual system is a representation of bound features from within and across the two visual streams (ventral and dorsal). However, it is not known at which stage of visual processing these features are bound together. Neurons within the middle temporal area (MT) possess the ability to process both local (component) and global (pattern/plaid) motion (Britten, Shadlen, Newsome, & Movshon, 1992; Recanzone, Wurtz, & Schwarz, 1997) and are able to determine global motion direction apart from other randomly moving stimuli. This suggests that the inputs to MT are integrated in order to determine the global motion of several moving objects. Binding these features together makes area MT suitable for determining the motion directions of multiple objects within the same spatial location (Adelson & Movshon, 1982; Stoner & Albright, 1992, 1996) and in turn allows for the segmentation of a visual scene into objects and surfaces (Snowden, Treue, Erickson, & Andersen, 1991). 

In spite of these characteristics that allow MT to process superimposed global motion, this type of motion has been shown to produce a perceptual illusion known as direction repulsion (Braddick, Wishart, & Curran, 2002; Curran & Benton, 2003; Hiris & Blake, 1996; Marshak & Sekuler, 1979; Mather & Moulden, 1980). In this case, the directions of motion of two superimposed surfaces are misjudged perceptually. Observers perceive the directions of motion as being further away from each other, for example, repulsed from 4° to 20° away from each surface's real direction (Braddick et al., 2002; Marshak & Sekuler, 1979). In the classic direction repulsion paradigm, the surfaces are identical except for the direction in which they are moving. This means that, first, the local motion of the dots in each surface must be calculated before they can be segmented into two surfaces and then the overall direction of each surface can be processed and a decision threshold reached. However, the addition of a second surface feature, making the surfaces more distinct from one another, should provide additional information that could be used to reduce the competition between the surfaces' directions and attenuate the repulsion. And, in fact, this is what occurs when the surfaces are different speeds (Curran & Benton, 2003; Marshak & Sekuler, 1979), or, in the case of superimposed gratings, when the surfaces are different spatial frequencies (Kim & Wilson, 1996). Stereoscopic viewing, producing a real depth difference between the two surfaces, does not reduce direction repulsion however (Hiris & Blake, 1996). This is thought to be because superimposed surfaces are already perceived as being at different apparent depths (Hiris & Blake, 1996) and therefore stereoscopic depth cannot be used as an additional feature to aid in segmenting the surfaces. 

Speed and direction, along with spatial frequency and depth, are all constituents of motion processing that occurs within the dorsal stream. Previously (Perry & Fallah, 2012), we tested whether the integration of a ventral stream feature, such as color, could also alter direction perception. Color is a motion-irrelevant feature, and neurons in area MT are not known to be color sensitive (Maunsell & Van Essen, 1983). In order for color to alter direction perception, color information from the ventral stream would have to be integrated (or bound) to the surface before or at the time of motion processing in area MT. We found that segmenting two superimposed surfaces by color did not alter direction repulsion but, surprisingly, did significantly decrease processing time. This shows that color is not bound to motion before global direction processing in area MT occurs. However, color does affect processing time suggesting that color may affect decision-making in areas downstream of area MT (Huk & Shadlen, 2005; Hussar & Pasternak, 2013; Shadlen & Newsome, 1996, 2001; Zaksas & Pasternak, 2006). Therefore, color and motion are bound after global motion processing in area MT. 

Based on those findings, we hypothesized that all segmentation cues bound to an object should speed up decision making about features of that object. Ventral stream features such as color showed just such an effect (Perry & Fallah, 2012). In the current study, we investigated whether speed segmentation cues would also reduce processing time. This is important to determine as motion processing in area MT is based on the conjunction of speed and direction, and thus the features are potentially linked before being integrated into the object's representation. We expect that, due to the conjunction, differences in speed will affect direction repulsion. However, that by itself should not reduce processing time. If we find that differences in surface speeds also produce reductions in processing time, then it suggests that speed information is also treated as a feature independent of direction at a later stage of decision making, similar to the effects of color differences. Alternatively, no changes in processing time would occur if velocity (the conjunction of speed and direction) is the feature bound into the object representation used by the decision-making circuitry. 

Methods

Participants

Twelve naive participants (ages 18–23, 5 female) completed the 3/6:unicolor paradigm and an additional set of 12 participants (ages 18–39, 10 female) completed the 6/6:unicolor paradigm. All participants provided informed consent, had normal or corrected-to-normal visual acuity and none tested positive for color blindness using Ishihara color plates. Ethics approval was provided by the York University Human Participants Review Committee. 

Procedure

Experiments were performed in a darkened, quiet room. Participants sat 57 cm from a computer monitor (21 in. Viewsonic, 1028 × 1024 resolution, 60 Hz) with their head positioned and stabilized on a headrest (Headspot, UHCOtech, Houston, TX). Participants wore a head-mounted infrared eye tracker (Eyelink II, SR Research Ltd., 500 Hz, Mississauga, ON, Canada) monitoring the left eye. Superimposed random dot kinetograms (RDKs) were created using MATLAB (MathWorks, Natick, MA) and experimental control was maintained using Presentation (Neurobehavioral Systems, Berkeley, CA) software. 

Each trial commenced with the participant fixating a white cross (Figure 1) centered on a black screen. 200 ms later a circular aperture appeared in the lower right quadrant containing two superimposed surfaces containing 100% coherent RDKs (white: 122 cd/m², dot size = 0.04°, aperture size = 5°, dot density = 1.54 dots/degree²). The experimental paradigm is the same as used previously (Perry & Fallah, 2012) except that instead of varying surface color we varied surface speed in the current study. In the 6/6:unicolor condition both surfaces moved at 6°/s. In the 3/6:unicolor condition, one surface of dots moved at 3°/s and the other at 6°/s. 

Figure 1

 

Experimental paradigm. In each condition, a trial is initiated with the appearance of a fixation point in the middle of the screen. When fixation has been maintained for 200 ms, the superimposed RDKs are then presented in the lower right quadrant of the screen. Once the stimulus has disappeared, a circular outline (the response circle) is presented at the same location as the stimulus. Participants made two clicks on the response circle indicating the directions in which the two surfaces were moving. In the 6/6:unicolor condition, the surfaces both move at 6°/s and in the 3/6:unicolor condition one surface moves at 3°/s and the other at 6°/s.

The surfaces moved in 12 directions relative to both the vertical and horizontal axes (±2°, 6°, and 10° from either up or down and left or right). All directions appeared with equal frequency creating differences between the two directions that ranged from 70° and 100°. If fixation was broken before or during stimulus presentation, the trial was aborted and randomly replaced. After the stimulus disappeared, a circular outline (the response circle) replaced the aperture. The participant was required to make two mouse clicks on the response circle indicating the directions in which the two surfaces were moving. 

Stimulus duration was varied using a staircase design (Perry & Fallah, 2012). A block consisted of eight trials at a given stimulus duration (initial duration: 2000 ms). If performance (the ability to get both directions correct) in a given block was ≥87.5% (7/8) the stimulus duration in the next block was decreased. When performance fell below this threshold, indicating the stimulus duration was not long enough to correctly determine both directions, stimulus duration in the subsequent block was increased. The staircase had two stages. In the first, stimulus duration increased or decreased by 500 ms step sizes. Upon reaching a double reversal, stage two commenced in which the step size was 100 ms. The staircase ended when a second double reversal occurred. This allowed us to estimate the time needed to correctly process both directions of motion to within ±50 ms. 

Data analysis

Correct responses were defined to allow for repulsion as in the previous study (Perry & Fallah, 2012): responses that fell within a range that extended from halfway between the two directions to 45° away from each real direction. A correct trial was defined as being any trial in which the participant determined both directions of motion within the ranges described above. Direction repulsion was calculated as the perceived direction minus the real direction of motion, so that positive values were indicative of direction repulsion. Means were calculated for both direction repulsion and processing time and independent t tests were used to assess any statistical differences between the conditions. When comparing the data to the 3/3:unicolor condition from the previous study, one-way ANOVAs with Tukey post hoc tests to control for multiple comparisons were utilized. The data was analyzed using MATLAB and SPSS (SPSS Inc., IBM, Armonk, NY). 

Results

Previous work has found that increasing the strength of surface segmentation, using features processed within the dorsal stream, improved perception of direction (Kim & Wilson, 1996; Marshak & Sekuler, 1979). However, we previously determined that increasing the strength of surface segmentation using a ventral stream feature did not affect direction perception but instead reduced processing time (Perry & Fallah, 2012). In this study we wanted to determine if increasing the strength of surface segmentation using a dorsal stream feature would similarly reduce processing time in addition to improving direction perception. From the results, we can then determine when different features are bound together. 

Direction repulsion

To determine how a difference in speed, 3/6:unicolor, affects direction repulsion compared to equal speeds, 6/6:unicolor and 3/3:unicolor from the previous study (Perry & Fallah, 2012), we performed a one-way ANOVA and post hoc Tukey HSD tests. We found a significant effect of surface speeds on direction repulsion (Figure 2A, F(2,33) = 4.51, p = 0.019). Increasing the speed of both surfaces, in the 6/6:unicolor condition (DR: 10.10° ± 0.74 SEM), significantly reduced direction repulsion when compared to the 3/3:unicolor condition (DR: 13.93 ± 1.38 SEM, p = 0.027), consistent with increased speed of motion reducing direction repulsion (Braddick et al., 2002). If there were no additional effect of speed segmentation on the attenuation of direction repulsion, then the repulsion in the 3/6 condition should fall between the repulsion in the 3/3 and 6/6 conditions, as the sum of the repulsion produced by one 3°/s surface (13.93°/2 = 6.97°) and one 6°/s surface (10.10°/2 = 5.05°) estimates a 12.02° repulsion. However, the repulsion in the 3/6 condition (10.47°) was significantly less from that seen in the 3/3:unicolor (previous study) condition (DR: 13.93° ± 1.38 SEM, p < 0.05) and was nearly identical to that in the 6/6 condition (10.10°, Figure 2A, p = 0.961). Therefore, speed segmentation likely provided additional attenuation of direction repulsion above that produced by an increase in the speed of one surface. Next we addressed the question of interest: Does speed segmentation affect processing time? 

Figure 2

 

Direction repulsion and processing time. These graphs combine results from the current study and Perry & Fallah, 2012. (A) Direction repulsion in the 3/6:unicolor (10.47° ± 0.74 SEM) and 6/6:unicolor (10.10° ± 0.74 SEM) conditions was not significantly different. Repulsion in these two conditions was significantly less than in the 3/3:unicolor (13.93° ± 1.38 SEM) condition. (B) Processing time in the 3/6:unicolor (483 ms ± 80.09 SEM) condition was significantly less than in the 6/6:unicolor (950 ms ± 132.57 SEM) and the 3/3:unicolor (1488 ms ± 208.5 SEM) conditions. Errors bars represent SEM.

Processing time

When we compared the time needed to process both surfaces correctly (processing time, Figure 2B) in the 6/6:unicolor and 3/6:unicolor conditions, we found that speed segmentation afforded a significant, t(22) = 3.013, p = 0.006, advantage. The average time needed in the 6/6:unicolor condition, 950 ms (±132.57 SEM), was reduced by nearly 500 ms (467 ms) when the surfaces were different speeds (3/6:unicolor = 483 ms ± 80.10 SEM). When compared to the results from our previous study, we found a significant effect of surface speed on processing time, F(2,33) = 11.23, p < 0.001). Segmenting the surfaces by increasing the speed of one surface (3/6:unicolor condition) significantly reduced processing time by approximately 1000 ms when compared to the slower speed 3/3:unicolor condition (1488 ms, ±208.54 SEM, p < 0.001). However, increasing the speed of both surfaces (6/6:unicolor condition) reduced that benefit by half from about 1000 ms to approximately 500 ms, (p = 0.042). Therefore, task-irrelevant speed segmentation cues reduce the processing time needed for direction judgments. 

Discussion

Direction repulsion

Using the same experimental paradigm as used previously (Perry & Fallah, 2012) we were able to determine how the speed of the surfaces affect direction repulsion under a number of conditions: two matching speeds (3/3:unicolor, 6/6:unicolor), and a speed segmentation condition where the speeds differed (3/6:unicolor). Consistent with previous literature, we found in the current study that differences in surface speed attenuated direction repulsion (Curran & Benton, 2003; Marshak & Sekuler, 1979). Also consistent with prior research (Braddick et al., 2002; Curran & Benton, 2003), we found that increasing the speed of both surfaces (6/6:unicolor) similarly reduced direction repulsion, likely due to increases in speed strengthening the representation of motion information (Maunsell & Van Essen, 1983; Palmer, Huk, & Shadlen, 2005). With the addition of speed differences or increase in the speed of both surfaces, attention of direction repulsion reached its limit: approximately 10° for two direction judgments (Braddick et al., 2002) or about 4° for a single direction judgment (Curran & Benton, 2003). In comparison, differences in surface color do not attenuate direction repulsion (Perry & Fallah, 2012). Therefore, direction repulsion is modulated by features processed within the dorsal stream, such as speed and spatial frequency, but not by features processed within the ventral stream, such as color. This suggests that direction repulsion occurs prior to color and motion being bound into an object representation. Thus, it is likely that direction repulsion is driven by a local circuit in area MT prior to forming an object representation that includes ventral stream information. 

Neural circuitry: Direction repulsion

Direction repulsion was originally described as arising from mutual inhibition (Marshak & Sekuler, 1979; Mather & Moulden, 1980) where the neurons responding to one direction inhibit the neurons responding to the other direction. The amount of mutual inhibition also varied by the difference in directions, with repulsion decreasing as the difference increased (Marshak & Sekuler, 1979; Mather & Moulden, 1980). We propose a mutual inhibition circuit wherein each direction inhibits the other based on the overlap in tuning between the neurons representing each direction (Figure 3). Figure 3A depicts a population of area MT neurons with preferred directions of −15°, 0°, and +15° all of which respond to rightward motion (0°). The population tuning curve to the right of the polar plot depicts how the responses are integrated to determine the direction of motion (peak population response). When a second surface is added moving downwards (270°), the responses to that direction proportionally inhibit the first direction's responses based on the amount of overlap in the tuning curves. The population tuning curve is reduced but more importantly, the peak direction is shifted away; that is, it is repulsed (Figure 3B). This model is supported by the following aspects. First, as the angular difference between the directions increases, the overlap in tuning decreases which reduces the repulsion as was previously found (Marshak & Sekuler, 1979; Mather & Moulden, 1980). Second, direction tuning, like orientation tuning, is wider at oblique angles and sharper on the cardinal axes (Coletta, Segu, & Tiana, 1993; Gros, Blake, & Hiris, 1998; Hiris & Blake, 1996). When one direction is on a cardinal axis, the range of angles that produce repulsion is more limited (Marshak & Sekuler, 1979; Mather & Moulden, 1980) compared to when both directions are oblique (Braddick et al., 2002; Perry & Fallah, 2012). 

Figure 3

 

Uni- and multidimensional mutual inhibition. (A) Depicts individual neurons and population tuning curves for rightward motion. (B) The addition of a second surface suppresses the neuronal responses and shifts the population tuning away from the real direction due to inhibition whose strength is based on the overlap between the tuning curves. (C–F) Multidimensional tuning curves for speed and direction. Above each polar plot is a depiction of the direction tuning overlap for comparison. The greater the size of the overlapping region, the greater the mutual inhibition. (C) When all motion features are identical except for direction, multidimensional tuning is reduced to direction alone, or is unidimensional. (D) Segmenting the surfaces by an additional motion feature (such as speed) changes the population of neurons engaged in mutual inhibition, thus diminishing the area of overlap and reducing direction repulsion. Note that the overlap in the direction dimension (curves above) is no different than when the speeds are the same (C). (E) Attention to one of the surfaces suppresses the influence of the second surface. This reduction in gain of one surface shrinks the population response underlying that surface, which in turn reduces the overlap between the two causing a reduction in direction repulsion. (F) The addition of a ventral stream (color) feature difference, unlike speed, did not influence direction repulsion, and thus was not a feature dimension in the mutual inhibition circuit.

We further propose that the mutual inhibition circuit is based not only on the overlap of direction tuning between neurons, but more so on the overlap of multidimensional tuning across conjunctions of motion features such as speed, spatial frequency, and direction selectivity (Albright, 1984; Lagae, Raiguel, & Orban, 1993; Maunsell & Van Essen, 1983; Perrone & Thiele, 2001). When other motion features are identical between the two surfaces, the multidimensional tuning is reduced to directionality alone (Figure 3C). However, adding a second distinguishing motion feature, such as speed, would reduce the overlap between the multidimensional tuning curves, thus reduce mutual inhibition and direction repulsion (Figure 3D). As speed and spatial frequency are features that form conjunctions with direction tuning in the dorsal stream, this model supports the reduction in direction repulsion seen with differences in speed (current study; Marshak & Sekuler, 1979) or spatial frequency (Kim & Wilson, 1996). Finally, this model also describes the effects that attending to one surface has on direction repulsion. Attention to speed or luminance changes in one superimposed surface reduced direction repulsion but dividing attention across both surfaces did not (Chen, Meng, Matthews, & Qian, 2005). The authors suggest that the results when attending to one surface can be explained based on feature-similarity gain (Martinez-Trujillo & Treue, 2004; Treue & Maunsell, 1999) in which attention enhances the representation of the attended feature and simultaneously reduces the influence of the unattended feature. Since the features in question are dorsal stream features coprocessed by directionally selective cells in area MT, the effect of attending to one surface while suppressing the other would be to reduce the gain of the suppressed surface and thus reduce the overlap for mutual inhibition (Figure 3E). This would produce the attenuation in direction repulsion that was seen (Chen et al., 2005). Finally, as color differences did not reduce direction repulsion, color is not a feature dimension used by the mutual inhibition circuitry. The multidimensional tuning for mutual inhibition works on dorsal stream, not ventral stream, features (Figure 3F). 

Processing time

Having previously found that color segmentation did not affect direction discrimination but did increase the speed of processing, we investigated whether speed segmentation also reduces the processing time needed to make direction discriminations. There is a time cost associated with the integration of features over the processing of single features (Bartels & Zeki, 2006; Bodelón, Fallah, & Reynolds, 2007). Also, adding additional features increases the perceptual load, which generally slows processing (Lavie, 1995) Thus, further segmenting the surfaces by adding irrelevant features, such as speed or color differences, requires binding and should take longer than processing direction alone. However, we have found that there is a substantial advantage to be had by integrating features when the end result is to increase segmentation between superimposed surfaces. Using differences in surface speed (current study) and color (Perry & Fallah, 2012) we have shown that the time needed to process the direction of two superimposed surfaces can be reduced by over 500 ms. Therefore, the integration of features within the dorsal stream (speed and direction), where features are often coprocessed by neurons (Gross, Bender, & Rocha-Miranda, 1969; Holcombe & Cavanagh, 2001; Maunsell & Van Essen, 1983), and binding of features between the ventral and dorsal streams (color and direction) both produce a significant advantage in how quickly the information is processed. 

While increasing the speed of one surface to 6°/s produces speed segmentation (vs. 3°/s), increasing both surfaces' speeds to 6°/s does not. If in the speed segmentation (3/6:unicolor) condition, the reduction in processing time is solely due to increasing the speed of the one surface, then increasing the speed of both surfaces should reduce processing time further, or if processing time is already at its lower limit, produce the same processing time advantage. Instead, we found that increasing both surfaces' speeds to 6°/s reduced the processing advantage. Differences in speed provide a greater advantage to direction judgments than just moving at faster speeds. Note that there was still a (smaller) advantage for the matched faster speeds (6/6:unicolor) over the matched slower speeds (3/3:unicolor). An equivalent increase in speed raised the response rates of area MT neurons (Maunsell & Van Essen, 1983) presumably increasing the strength of the motion representation. As others have shown reduced reaction times from increasing stimulus strength by luminance (Pins & Bonnet, 1996) or motion coherence (Palmer et al., 2005), increasing motion strength by increasing surface speed should also reduce reaction times. Our results show how reduced processing time would underlie these faster reaction times. 

Neural circuitry: Processing time

We propose that the large decrease in processing time that occurs with increases in surface segmentation by additional features is most likely due to speeding up decision making (see Perry & Fallah, 2012). Motion direction is processed in area MT (Albright, 1984; Mikami, Newsome, & Wurtz, 1986; Newsome & Pare, 1988; Salzman, Murasugi, Britten, & Newsome, 1992) and passed forward to frontal and parietal areas, which can accumulate the direction information in order to reach a decision threshold (Huk & Shadlen, 2005; Hussar & Pasternak, 2013; Shadlen & Newsome, 1996, 2001; Zaksas & Pasternak, 2006). When two surfaces are identical except for direction of motion, each surface's direction information interferes with the processing of the other surface's direction, creating a “noisy walk” toward the decision threshold (accumulator model, Palmer et al., 2005). By introducing differences in color (Perry & Fallah, 2012) or speed (current study), the objects become more distinct from each other, providing additional features through which the direction information can be separated. Filtering out the input from the other surface would reduce the noise in the walk to threshold, increasing the slope of information accumulation. Thus, the decision threshold would be reached sooner resulting in decreased processing time. 

Conclusion

Irrelevant speed segmentation cues reduce the processing time required to make direction judgments. Color segmentation cues also reduce the processing time required to make direction judgments (Perry & Fallah, 2012). However, only speed affects direction processing as measured by changes in magnitude of direction repulsion, an illusion linked to a local mutual inhibition circuit within area MT. Therefore, motion processing integrates speed and direction prior to global motion processing. The output of global motion processing feeds forward to decision-making areas, where color segmentation cues, as well as speed, reduce processing time. Therefore, by this stage the object representation includes ventral (color) and dorsal (speed and direction) information. Thus, the integration of features within and across the streams occurs at different stages of processing along the visual hierarchy. 

Acknowledgments

The research was funded by an NSERC Discovery Grant to Mazyar Fallah and an NSERC Alexander Graham Bell Canadian Graduate Doctoral Scholarship to Carolyn J. Perry. 

Commercial relationships: none 

Corresponding author: Carolyn J. Perry. 

Email: [e-mail]. 

Address: School of Kinesiology and Health Science, Centre for Vision Research, York University, Ontario, Canada.