Transparent surface segregation enables visual feature binding in rapidly alternating displays

Abstract

Visual feature binding—the mechanism by which our typically coherent and unified perceptual experience arises from distributed neural representations—is the source of much intrigue in the neuroscience of perception. Surprisingly, feature binding can occur in rapidly alternating displays of color–orientation combinations (e.g., rightward–orange, leftward–blue). However, we found that when the angular separation between orientations is reduced, binding is selectively impaired at temporal alternation frequencies around 5 Hz. To isolate the mechanisms involved, we devised a novel display in which color–orientation conjunction information was distributed temporally over two checkered stimuli and was perceptually discriminable only within an intermediate range of temporal frequencies (7.5–15 Hz). We propose that accurate color–orientation judgments at frequencies exceeding 5 Hz depend on the rapid formation of persistent surface representations that can be accessed by binding mechanisms, circumventing the latter's relatively low temporal resolution.

Introduction

Within the visual system, basic features such as color and orientation are encoded by the activity profile of populations of neurons (Zeki, 1978; Zeki et al., 1991). While many neurons in the early visual cortex appear to be selective for multiple features (Burkhalter & Van Essen, 1986; Johnson, Hawken, & Shapley, 2008; Leventhal, Thompson, Liu, Zhou, & Ault, 1995), a wealth of behavioral evidence indicates that a binding problem nonetheless exists (Clifford, 2010; Fujisaki & Nishida, 2010; Holcombe, 2009; Moradi & Shimojo, 2004; Quinlan, 2003; Treisman, 1996; Treisman & Schmidt, 1982; Wu, Kanai, & Shimojo, 2004). The visual system must take into account both the spatial location and the temporal coincidence of features in order to accurately represent objects comprising a combination thereof. This process of feature binding is considered to be relatively slow, necessitating multiple stages of processing (Holcombe, 2009; Treisman, 1996). 

Surprisingly, Holcombe and Cavanagh (2001) reported a high temporal resolution for feature binding. They used a stimulus that rapidly alternated over time between two orthogonally oriented and differently colored square-wave gratings in the same spatial location (Figure 1a; Bodelón, Fallah, & Reynolds, 2007; Holcombe, 2001, 2009; Holcombe & Cavanagh, 2001; Suzuki & Grabowecky, 2002). Such an arrangement elicits a temporal transparency illusion—the somewhat paradoxical phenomenon whereby each grating appears simultaneously distinct and transparent through perceptual segregation despite an absence of static transparency cues (Holcombe, 2001). Under these conditions, the combination of color and orientation features belonging to each surface can be accessed simultaneously and remains readily distinguishable, even at temporal alternation frequencies up to 19 Hz (Holcombe & Cavanagh, 2001). 

Figure 1

 

Stimulus design and results for Experiment 1. (a) An orange square-wave grating tilted in one direction was temporally alternated at the same spatial location with a blue grating tilted in the opposite direction. Gratings were presented at one of five angular separations (90° in this example; shown in purple). (b) Orange and blue gratings were calibrated such that they summed physically to an achromatic plaid (Holcombe & Cavanagh, 2001) wherein the color–orientation pairing information was lost. The plaid's appearance was independent of the color–orientation pairing. (c) Mean color–orientation conjunction discrimination across subjects (n = 6) as a function of both the angular separation and the temporal alteration frequency. Error bars denote ±1 SEM. See Supplementary Movies S1 and S2 for demonstrations of this stimulus.

On the basis of this high temporal limit, Holcombe and Cavanagh (2001) inferred that simultaneously presented feature pairs must be processed together and early in the visual system (also see Blaser, Papathomas, & Vidnyánszky, 2005; Favreau, Emerson, & Corballis, 1972). Otherwise, the temporal averaging performed by later visual processes would mask the correct feature pairing (Holcombe & Cavanagh, 2001). However, if feature binding occurs at an early stage of processing, this must be reconciled with evidence suggesting that binding has a low temporal resolution (Holcombe, 2009; Quinlan, 2003; Treisman, 1996; Treisman & Schmidt, 1982) and that color and orientation features themselves are processed in a way that gives rise to a perceptual color–orientation asynchrony (Clifford, Arnold, & Pearson, 2003; Moutoussis & Zeki, 1997). 

Previous studies of the binding of color with motion in transparent-motion displays offer the suggestion that the accurate feature binding observed under conditions of temporal transparency is instead tied to the generation of persistent surface representations (Clifford, Spehar, & Pearson, 2004; Moradi & Shimojo, 2004; Suzuki & Grabowecky, 2002; Vigano, Maloney, & Clifford, 2014). Here we propose a potential mechanism by which feature conjunctions can be identified through targeted feedback from higher visual areas (Bouvier & Treisman, 2010; Clifford, 2010; Di Lollo, Enns, & Rensink, 2000; Hochstein & Ahissar, 2002; Juan & Walsh, 2003). This is further examined in the General discussion. In a rapidly alternating stimulus display, the process of assigning the correct features to the respective surfaces does not appear to be completely resolved by the early visual system alone. The question remains as to how visual features are first associated with the correct surface representation. We suggest that attentional selection of a single feature (e.g., a rightward-tilted orientation) can enhance the responses of the population of neurons selective for this orientation. Included in this population are “double-duty cells” tuned to both color and orientation (Burkhalter & Van Essen, 1986; Gegenfurtner, 2003; Gegenfurtner, Kiper, & Fenstemaker, 1996; Gegenfurtner, Kiper, & Levitt, 1997; Johnson et al., 2008; Leventhal et al., 1995; Navon, 1990; Shipp, Adams, Moutoussis, & Zeki, 2009; Tamura, Sato, Katsuyama, Hata, & Tsumoto, 1996). Feedback targeted to boost the response of double-duty cells selective for this orientation will similarly enhance the response to the associated color, allowing the correct pairing of orientation and color to be decoded from the response profile of the population of double-duty neurons. 

Thus, the rapid nature of surface segregation (Møller & Hurlbert, 1996; Sajda & Finkel, 1995) may result in accurate conjunction perception that appears to have a high temporal resolution (Moradi & Shimojo, 2004). Here, we explored this idea in four experiments investigating the interrelationship between surface segregation and the binding of color with orientation. We first applied a straightforward modification to the temporally alternating colored grating stimulus used by Holcombe and Cavanagh (2001). Namely, we varied the angular separation between the gratings to test the surface-based limits of the temporal transparency illusion, given that the angular separation between orientations is an important surface segregation cue (Holcombe & Cavanagh, 2001; Kawabe & Miura, 2004; Nothdurft, 1991; Watanabe & Cavanagh, 1996). Subjects discriminated the color–orientation conjunction in a psychophysical “binding” task. The angular separation and temporal alternation frequency of the gratings interacted to reveal a temporally bandpass impairment in conjunction discrimination centered around 5 Hz. Experiments 2 and 3 were designed to clarify the extent to which the perception of the stimuli as transparent, superimposed surfaces influences conjunction discrimination. In Experiment 4, a novel checkered stimulus was used where reliable pairing of color with orientation was possible only within a narrow range of rapid temporal frequencies. This allowed us to further isolate the mechanisms involved and their dependence on a high temporal alternation frequency. Together, the results provide new insights into the temporal dynamics of color–orientation binding. 

Experiment 1: Discrimination of feature conjunctions using colored, oriented gratings

In their original study, Holcombe and Cavanagh (2001) used colored square-wave gratings that were oriented orthogonally to one another (see also Bodelón et al., 2007; Holcombe, 2001; Suzuki & Grabowecky, 2002). We reasoned that narrowing this angular separation might cause the rapid color–orientation binding normally possible in temporal transparency to break down, as a smaller separation between the gratings should impair the perception of each grating as a distinct surface (Nothdurft, 1991; Watanabe & Cavanagh, 1996). 

Subjects

Six experienced psychophysical subjects (four males; age range = 22–29 years), including two of the authors and four naïve subjects, participated in Experiment 1. All had normal or corrected-to-normal visual acuity, had normal trichromatic color vision, and were free of psychiatric or neurological illness. For the experiments reported here, all experimental protocols were approved by the University of Sydney Human Research Ethics Committee, and informed consent was provided by all subjects prior to the experiment. 

Apparatus

For this and all other experiments, subjects sat at a viewing distance of 57 cm from a gamma-corrected ViewSonic Graphics Series G90f cathode ray tube monitor (ViewSonic, Brea, CA) (36 cm × 27 cm) with a vertical refresh rate of 60 Hz and a resolution of 1024 × 768 pixels. Stimuli were generated through Matlab (R2010a 7.10; Mathworks, Natick, MA) and the Psychophysics Toolbox 3 (Brainard, 1997; Pelli, 1997) on a personal computer with an Intel Core i7-2600, 3.4 GHz processor, and AMD Radeon HD 6450 display adapter. Experiments were run in a lightproof and soundproof testing booth. 

Visual stimuli

The grating display contained two temporally alternating square-wave gratings (Holcombe & Cavanagh, 2001) with a spatial frequency of 0.84 cycles per degree of visual angle (Figure 1a). Gratings were presented at one of five angular separations: 5°, 10°, 15°, 20°, or 90° (orthogonal). For example, a 5° angular separation saw each grating rotated by ±2.5° from vertical. Gratings were presented within an annular region (inner radius = 2.21°; outer radius = 12.08°) against a mean luminance gray background (26 Cd m⁻²). Contrast at the edges of the annulus was ramped with a raised cosine profile of 0.84°. Gratings were either blue (CIE: x = 0.24, y = 0.28) or orange (CIE: x = 0.33, y = 0.36) in color. For both, one spatial half-cycle was light blue or orange (29 Cd m⁻²), whereas the other spatial half-cycle was dark blue or orange (23 Cd m⁻²). 

Colors were defined using DKL space (Derrington, Krauskopf, & Lennie, 1984) and calibrated individually for each subject prior to the experiment such that all combinations of blue and orange summed subjectively to a shade of gray (CIE: x = 0.28, y = 0.31) in order to avoid discrimination biases due to imbalanced color saturation (Holcombe & Cavanagh, 2001). When physically summed, the gratings produced an achromatic plaid such that the particular color–orientation combination could not be resolved (Figure 1b). Subjective equiluminance between pairs of light and dark colors was also calibrated using a minimum flicker paradigm (Walsh, 1953). The center of the annulus contained a small fixation cross (0.4° × 0.4°) encircled by a black-and-white fixation ring (diameter = 1.4°). 

Design and procedure

Subjects performed a color–orientation binding task in which they reported the tilt (leftward or rightward) of the orange grating (Figure 1a; also see Supplementary Movies S1 and S2). In this task, we used a 5 angular separation (5°, 10°, 15°, 20°, and 90°) × 8 temporal alternation frequency (1.25, 2.5, 3.75, 5, 7.5, 10, 15, and 30 Hz) within-subject factorial design (Figure 1a). Temporal alternation periods were chosen such that they divided evenly into 60 Hz. In all conditions, the total stimulus duration was 800 ms, including a 250-ms raised cosine contrast ramp at the beginning and end of each trial to eliminate onset and offset transients. 

In each experimental run of the feature binding task, each possible combination of angular separation with temporal frequency was repeated eight times (randomly interleaved). Subjects performed five repeat runs for a total of 40 trials per condition. Furthermore, trials within runs were counterbalanced for onset frame and color–orientation pairing. Prior to performing the feature binding task, subjects completed a practice run with auditory feedback. During the actual experiment, subjects reported the tilt (leftward or rightward) of the orange grating in an unspeeded manner on each trial using a standard keyboard while maintaining fixation on the central cross. 

Conjunction discrimination was coded as the proportion of correct responses over the duration of the experiment. These data were subjected to a two-way repeated measures analysis of variance (ANOVA) with subjects as a random factor and angular separation and temporal alternation frequency as fixed factors. The Greenhouse–Geisser correction was applied to the degrees of freedom in evaluating F ratios to guard against potential violations of sphericity. Where interactions or main effects were significant in the ANOVA, we report the outcomes of planned, Bonferroni-corrected polynomial contrasts up to a degree of 3 (cubic). 

Results and discussion

The aim of Experiment 1 was to identify the role of surface segregation in supporting accurate conjunction discrimination at high temporal alternation frequencies. Overall, the pattern of results suggests that a combination of cues important in surface segregation (angular separation and temporal alternation frequency; Clifford, Spehar, et al., 2004; Moradi & Shimojo, 2004; Nothdurft, 1991; Watanabe & Cavanagh, 1996) is needed to facilitate accurate conjunction discrimination at high alternation frequencies. 

Across temporal alternation frequency, an increase in conjunction discrimination was associated with an increase in angular separation; two-way repeated measures ANOVA with planned polynomial contrasts, linear trend, F(1, 5) = 47.32, p < 0.001 (Figure 1c). We suggest that this result is more likely due to a weakened surface segregation percept than to the higher task difficulty associated with the discrimination of similar orientations. Experiments 2 and 3 were designed to further verify this hypothesis. 

Across all angular separations, the selective impairment in conjunction discrimination at 5 Hz reveals a transition in the way that binding occurred; main effect of temporal frequency: linear trend, F(1, 5) = 43.79, p = 0.001, and cubic trend, F(1, 5) = 17.19, p = 0.009. An increase in alternation frequency from 1.25 to 5 Hz corresponded with a decrease in conjunction discrimination. At very low alternation frequencies, feature binding can (in principle) occur within a single presentation of a conjunction stimulus. An increase in frequency necessitates a shorter grating presentation that should thereby increase task difficulty due to the limited time by which a single conjunction stimulus can be sampled by the visual system. However, we found that conjunction discrimination improved from 5 to 10 Hz—the range of frequencies facilitating temporal transparency (Holcombe, 2001). In this range, the attributes of each grating are temporally integrated at the level of surface representations across the full trial duration. This creates stable representations of each grating in the form of perceptually transparent surfaces (Suzuki & Grabowecky, 2002). Access to these representations over a period exceeding individual grating presentations provides a sufficient temporal window within which feature binding can act (Clifford, Holcombe, & Pearson, 2004; Moradi & Shimojo, 2004; Vigano et al., 2014). However, from 10 to 30 Hz, surface segregation fails and conjunction discrimination falls to chance levels as gratings combine perceptually to form a gray plaid. Combined with the nonfoveal stimulus presentation, a high alternation frequency masked the colors present in the display in a way that produced a percept where individual gratings were no longer distinguishable. 

Temporal alternation frequency affected conjunction discrimination nonmonotonically for each angular separation; quadratic interaction, F(1, 5) = 69.35, p < 0.001. At frequencies around 5 Hz, when surface segregation is impaired by small angular separations (Nothdurft, 1991; Watanabe & Cavanagh, 1996), conjunction discrimination is similarly affected. This suggests that binding around alternation frequencies of 5 Hz and greater is reliant on surface segregation. From 2.5 to 5 Hz, a smaller angular separation interacted with the shorter grating presentation duration at these frequencies to interfere with the formation of orientation-defined surfaces, decreasing conjunction discrimination. This effect was most pronounced at 5 Hz, where discrimination was clearly lowered as a function of angular separation. However, as frequency increased from 5 to 30 Hz, the difference in conjunction discrimination across angular separation was reduced as temporal transparency appeared to mitigate the effects of small angular separations on conjunction discrimination. 

Thus, at high alternation frequencies, surface segregation appears to play a necessary role in feature binding. Furthermore, the impairment of performance at intermediate frequencies suggests that the temporal transparency illusion is driven by a number of surface cues—in this case, angular separation and temporal alternation frequency (Clifford, Holcombe, et al., 2004; Moradi & Shimojo, 2004; Nothdurft, 1991; Watanabe & Cavanagh, 1996). If accurate conjunction discrimination at, for example, 10 Hz was instead due to a rapid binding mechanism (Holcombe & Cavanagh, 2001), it would be expected that conjunction discrimination could only improve at lower alternation frequencies. Instead, a disproportionately larger decrease at 5 Hz than at 10 Hz was observed when angular separation was reduced, suggesting that feature binding can occur by extracting feature pairs from surface representations when these representations are available. In order to confirm the relationship between surface segregation and conjunction discrimination, Experiment 2 sought to determine the correspondence between the current stimulus manipulations and subjective judgments of surface segregation. 

Experiment 2: Subjective impressions of colored, oriented gratings

To identify the relationship between binding task performance and subjective impressions of surface segregation, we conducted an additional subjective judgment task where subjects reported whether the same stimuli used in Experiment 1 appeared transparent or not. This allowed us to directly relate the objective conjunction discrimination measures of the first experiment to subjective judgments of surface segregation. 

Subjects

Nine subjects participated in this experiment: two authors plus seven naïve subjects (four males; age range = 22–29 years). 

Visual stimuli

Stimuli identical to those used in Experiment 1 were presented (Figure 1a). 

Design and procedure

Subjects reported whether they saw the display according to one of three categories: sequential, transparent, or plaid. Subjects were instructed beforehand on the meaning of each response. Sequential was reported when subjects experienced the stimulus as discrete, successive presentations of the individual gratings. Transparent was reported when subjects perceived both gratings as simultaneously present but still individually distinguishable (Holcombe, 2001; Holcombe & Cavanagh, 2001; Suzuki & Grabowecky, 2002). Finally, plaid was reported when subjects were no longer able to distinguish individual gratings and instead viewed a single gray plaid (Figure 1b). When making judgments, subjects were instructed to ignore stimulus flicker or color saturation and instead focus on the sequential or simultaneous appearance of the gratings. 

The focus of this experiment was the range in which the experimental stimulus was subjectively perceived as transparent. Thus, data were coded and are presented here as the mean proportion of transparent responses for each condition. Sequential and plaid responses were combined because they represent the two extremes of the stimulus being perceived as nontransparent. 

Results and discussion

This experiment aimed to identify the relationship between the perceptual interpretation (Figure 2) and conjunction discrimination (Figure 1c) of the color–orientation stimulus. Across angular separations, reports of perceptual surface segregation of the stimulus rose most rapidly after 5 Hz and peaked at 10 Hz. (Note that this result coincides fairly well with those of experiment 2 in Holcombe, 2001.) Within the range of 5 to 30 Hz, the perception of transparency rose and fell in the same manner as the conjunction discrimination measured in Experiment 1; two-way repeated measures ANOVA with planned polynomial contrasts: main effect of temporal frequency, F(7, 56) = 4.91, p < 0.001; cubic trend, F(1, 3) = 58.28, p < 0.001. Below 5 Hz, stimuli were generally reported as sequential, while plaid was reported most often at 15 Hz and greater. The angular separation by alternation frequency interaction indicates that both stimulus attributes had an effect on surface segregation. Overall, the transition in perception from a sequentially presented display to transparent surfaces was identified at around 5 Hz, matching the alternation frequency where conjunction discrimination broke down across angular separation in Experiment 1. 

Figure 2

 

Results for Experiment 2. Mean proportion of transparent responses across subjects (n = 9) as a function of both the angular separation and the temporal alternation frequency. Error bars denote ±1 between-subjects standard errors. See Supplementary Movies S1 and S2 for demonstrations of this stimulus.

Across temporal frequency, angular separation appeared to have no effect on the transparency of the stimulus; nonsignificant main effect of angular separation, F(4, 32) = 1.23, p = 0.317. A significant interaction effect was present in the data, F(28, 224) = 2.77, p < 0.001. A quadratic interaction effect where differences in conjunction discrimination between angular separation first increased and then decreased across alternation frequency was not found, F(1, 8) = 3.11, p = 0.77. The observed interaction most likely stems from subtle effects in stimulus interpretation, which was not the main focus of this experiment. We hypothesized that the angular separation by alternation frequency interaction present in Experiment 1 would be mirrored in Experiment 2. While several studies have previously indicated a relationship between feature binding and perceptual surface segregation (Clifford, Spehar, et al., 2004; Moradi & Shimojo, 2004; Vigano et al., 2014), here we did not find evidence that angular separation differentially affected reported transparency. On inspection of the data, it appears that at 90° angular separation the proportion of transparent responses was not at ceiling, even at the peak of the curve, unlike at some of the other angular separations. We speculate that this result and the lack of a significant interaction may be in part due to perceptual rivalry, whereby the alternating gratings vie for perceptual dominance over time (Brascamp, Van Ee, Pestman, & Van Den Berg, 2005; Leopold & Logothetis, 1999; Mamassian & Goutcher, 2005). 

Together, Experiments 1 and 2 highlight two distinct temporal frequency ranges where accurate conjunction discrimination is possible (less than 5 Hz and around 10 Hz), suggesting a direct relationship between surface segregation and conjunction discrimination. Therefore, two further stimulus displays were devised in Experiments 3 and 4 in order to isolate the processes driving conjunction discrimination at each frequency range. 

Experiment 3: Discrimination of spatially segregated color–orientation conjunctions

Experiment 3 used a stimulus in which the color and orientation features were segregated spatially (e.g., Fujisaki & Nishida, 2010; Holcombe & Cavanagh, 2001; Karlsen, Allen, Baddeley, & Hitch, 2010). In such an arrangement, the temporal coincidence of the color and orientation pairings remains unchanged from Experiments 1 and 2. However, visual features that are not colocated cannot produce the impression of a single surface, and, as such, their accurate binding cannot be supported by the formation of a transparent surface representation. In Experiment 4, the converse situation was tested: Color and orientation features never co-occurred at a single point in time. Rather, they were grouped together only when perceived as transparent surfaces (i.e., under conditions of temporal transparency). 

Subjects

A subset of five of the six subjects who participated in Experiment 1 also took part in Experiment 3 (four males; age range = 22–29 years). 

Visual stimuli

The annulus in this experiment was divided along the horizontal meridian (Figure 3a). In one half a gray square-wave grating was displayed, while in the other a solid block of color was displayed. Gratings had physical properties that were identical to those in Experiment 1 (luminance, spatial frequency, and angular separation), but here they were gray. The block of color was either orange (CIE: x = 0.33, y = 0.36) or blue (CIE: x = 0.24, y = 0.28) with a luminance of 26 Cd m⁻² (the mean of the light and dark colors from the first two experiments). Both halves of the display alternated simultaneously at one of several temporal alternation frequencies. Each half was separated vertically by a 1.5° gap. The grating always alternated between left and right tilted, and the block of color always alternated between blue and orange. Orientation and color were randomly assigned to the upper or lower portion of the annulus on each trial. 

Figure 3

 

Stimulus design and results for Experiment 3. (a) A gray square-wave grating and a solid block of color temporally alternated in orientation and color, respectively. Subjects reported the color–orientation pairing. Gratings were again presented at one of five angular separations (90° in this example; shown in purple). (b) Mean color–orientation conjunction discrimination across subjects (n = 5) as a function of both the angular separation and the temporal alteration frequency. Error bars denote ±1 SEM.

Design and procedure

As in Experiment 1, subjects performed a color–orientation binding task in which they reported the tilt of the gray grating that was paired with the orange block of color. A 5 angular separation (5°, 10°, 15°, 20°, and 90°) × 8 temporal alternation frequency (1.25, 2.5, 3.75, 5, 7.5, 10, 15, and 30 Hz) within-subject factorial design was also used. Subjects performed five repeat runs for a total of 40 trials per condition. In addition to onset frame and color–orientation pairing, location (upper or lower) of orientation and color was counterbalanced. 

Results and discussion

In Experiments 1 and 2, we determined two different regimes supporting accurate conjunction discrimination: (a) at low temporal frequencies where feature binding could occur (in principle) within a single stimulus half-cycle and (b) within the higher range of frequencies supporting temporal transparency. In this experiment, we used a spatially segregated color and orientation stimulus in which temporal transparency was not possible in order to isolate color–orientation binding mechanisms from the perception of surface transparency. The results of this experiment are given in Figure 3b. Using a two-way repeated measures ANOVA, we found that accurate conjunction discrimination was possible for temporal frequencies below 5 Hz—main effect of alternation frequency: F(7, 32) = 25.31, p < 0.001—qualitatively matching the results of Experiment 1 (Figure 1c). However, from 5 Hz and beyond, conjunction discrimination remained at chance for all angular separations and did not systematically increase within the range of 7.5 to 15 Hz. Furthermore, there was no significant main effect of angular separation, F(4, 12) = 1.83, p = 0.24, or angular separation by alternation frequency interaction, F(28, 84) = 0.79, p = 0.52. The nonsignificant main effect of angular separation indicates that the discrimination of a left-tilted grating from a right-tilted grating was not a factor influencing task difficulty in Experiment 1. 

The spatially segregated stimulus used here provides results in line with traditional thinking on feature binding. That is, the identification of features and ensuring their temporal coincidence is a process with a relatively low temporal resolution (Fujisaki & Nishida, 2010; Holcombe, 2009; Treisman, 1998; Treisman & Gelade, 1980). Despite a high temporal alternation frequency, surface segregation was not possible here because color and orientation attributes were not spatially colocated (Holcombe, 2001), and thus conjunction discrimination did not increase at intermediate alternation frequencies as it did in Experiment 1. We demonstrate in Experiment 4 that this low resolution can be overcome if the gratings are perceived as separate, transparent surfaces. 

Experiment 4: Discrimination of temporally distributed color–orientation conjunctions

Experiment 3 examined color–orientation binding processes in a stimulus in which the two features were not conjoined at overlapping spatial locations but were present at the same point in time. In Experiment 4, we introduced a stimulus in which the color–orientation conjunction information was distributed over two check patterns. Here, color and orientation features were not paired either spatially or temporally but rather were distributed equally in a checkered fashion. Upon an extended inspection of a single check pattern, subjects would identify that the orange and blue squares are arranged such that there is no leftward or rightward tilt bias, producing chance-level discrimination (Figure 4b). Instead of relying on spatial and temporal coincidence, as was the case in Experiments 1 and 3, the color and orientation pairing here can be found only through the temporal integration of color across check patterns. Grouping the same color across both check patterns produces gratings identical to those previously used, thus revealing the associated orientation. The logic of this display is similar to that used in Holcombe (2001), who used a display separated into two halves with differing luminances in each half. When each half varied in luminance simultaneously at a high alternation frequency (14 Hz), subjects tended to group gratings by luminance rather than by their physical, temporal coincidence. In this way, the effect of transparent surface segregation on conjunction discrimination can be isolated from the low-resolution feature binding processes measured in Experiment 3. 

Figure 4

 

Stimulus design and results for Experiment 4. (a) Grating stimuli were identical to those used in Experiment 1. In this experiment, however, only angular separations of 90° were used. (b) Check stimuli were designed such that only grouping by color across time revealed the color–orientation pairing, as demonstrated in the micropattern portion. In both panels a and b, participants reported the tilt of the orientation (left or right) associated with orange as described in the methods. (c) As in Experiment 1, both grating and check stimuli summed physically to an achromatic plaid wherein the display type was masked and the color–orientation pairing information was lost. (d) Mean color–orientation conjunction discrimination across subjects (n = 5) for both grating and check displays at each temporal alteration frequency. Error bars denote ±1 SEM. See Supplementary Movie S3 for a demonstration of this stimulus.

Subjects

Five experienced psychophysical subjects (three males; age range = 22–46 years), including the three authors and two naïve subjects, participated in this experiment. 

Visual stimuli

Two types of stimuli were used here. The grating display was identical to the 90° condition in Experiment 1 (Figure 4a; see also Supplementary Movie S1). The check display contained the same color–orientation conjunction information as the grating display but was distributed temporally over two blue–orange checked stimuli (Figure 4b; see also Supplementary Movie S3). Unlike the grating display, the color–orientation conjunction in the check display was not available within a single half-cycle of the stimulus. However, through temporal integration of a rectified representation of the stimulus, color information can (in principle) be extracted and grouped to yield the associated orientation across a full stimulus cycle. If colors are grouped in this way, subjects would be left with the perception of coherent, oriented strips as per the grating condition. In this way, although the time-averaged information in the grating and check displays was the same (e.g., Figure 4c), the check stimulus required temporal transparency in order for the conjunction information to be extracted. 

Design and procedure

This experiment used a 2 display type (grating, check) × 8 temporal alternation frequency (1.25, 2.5, 3.75, 5, 7.5, 10, 15, and 30 Hz) within-subject factorial design. As in Experiments 1 and 3, subjects performed a color–orientation binding task for both grating and check displays. However, subjects' conjunction discrimination for the check display was a measure of how temporal integration of a rectified stimulus representation enabled the perception of a conjunction in the check stimulus. An accurately perceived check display would reveal no left or right tilt for each color by design, which was predicted to be the case at low alternation frequencies. However, rapid alternations between check displays may instead bias subjects' perception of the stimulus, but only if similar colors tended to be perceptually grouped over time as per surface segregation. Thus, for the check display, the measured proportion of correct conjunction discrimination responses was in fact a measure of how well subjects identified the spatial and temporal relationships between individual check patterns. Subjects performed five repeat runs for a total of 80 trials per condition. 

Results and discussion

In Experiment 4 we removed the temporal coincidence of color and orientation features in the check stimulus such that the conjunction information was not available within a single half-cycle of the stimulus (Figure 4b). Extraction of the feature conjunction necessitated the temporal integration of feature information. Thus, we were able to observe the temporal profile of the feature binding process acting on segregated, transparent surface representations. 

The blue squares in Figure 4d show the mean conjunction discrimination for the check display. A two-way repeated measures ANOVA indicated a significant linear interaction between the stimulus display type (perpendicular grating vs. check) and the temporal frequency, F(1, 4) = 182.31, p < 0.001, in addition to main effects of both stimulus display type, F(1, 4) = 53.23, p = 0.002, and temporal frequency, F(1, 4) = 28.23, p < 0.001. Similar to Experiment 1, at the highest frequency tested (30 Hz), the checks were indistinguishable from a plaid (Figure 1c) and conjunction discrimination fell to chance for both display types. Apart from this, reliable discrimination of the color–orientation conjunctions was possible in both the perpendicular grating and check displays, but only within an intermediate range of temporal frequencies for the checks (around 7.5–15 Hz). At the lowest frequencies tested (1.25–2.5 Hz), the conjunction could still be reliably discriminated in the grating display but not the check display. The results observed here reflect the time required by the visual system to identify and process both features individually. 

Here we compared the check display with an orthogonal set of gratings (e.g., the 90° angular separation in Experiment 1). The linear interaction reported here is not likely due to an artefact of the ceiling performance observed in the grating display type in this experiment (at all frequencies except 30 Hz). A repeated measures ANOVA between the check display and any of the angular separations from Experiment 1 also produced statistically significant linear interactions (e.g., 15° and checks), F(1, 4) = 43.20, p = 0.003. The comparisons between Experiments 1 and 4, where performance was not always at ceiling, and the comparison between display types within Experiment 4 both provide statistically significant linear interaction effects. These tests suggest that where there was a reduction in the difference between conjunction discrimination in the grating and check displays at intermediate frequencies, it was due to a similar underlying effect of alternation frequency. 

The check display viewed at an intermediate range of frequencies (7.5–15 Hz) enabled accurate reporting of conjunctions on average more than 80% of the time. Conjunction discrimination was possible only within the specific range of temporal frequencies supporting temporal transparency: Too slow an alternation would impede the integration of the feature information over time, while too rapid an alternation would cause the checks to fuse perceptually into a static plaid in the same way as the gratings (Figure 4c). Accurate conjunction discrimination within this range also indicated that the temporal transparency percept was one in which check displays were grouped by color instead of being perceived as two individual check patterns—a tendency associated with surface segregation (Watt & Phillips, 2000). That the grating and check data tended to converge as the alternation frequency increased suggests that similar strategies were being used in both displays. That is, a higher alternation frequency facilitated temporal integration, resulting in an extended window in which color–orientation pairs could be ascertained. In the General discussion, we speculate on the way in which this process may occur. 

General discussion

This study investigated the relationship between surface segregation and color–orientation feature binding. The complex temporal dynamics of the mechanisms of feature binding were revealed by manipulating several surface segregation cues. The results observed here suggest that accurate conjunction discrimination at high alternation frequencies is the result of feature binding accessing the information contained within persistent transparent representations. In effect, binding itself nevertheless remains a relatively slow process (Holcombe, 2009; Seymour, McDonald, & Clifford, 2009; Treisman, 1996). 

In Experiment 1, the nonmonotonic effect of angular separation on conjunction discrimination across temporal alternation frequencies revealed two distinct ranges of temporal frequencies in which accurate conjunction discrimination occurred. The results of this experiment suggest that there exists a transition in the way features are bound. In addition to binding within a single presentation (i.e., half-cycle of the stimulus) at lower frequencies, surface segregation appeared to aid the feature binding process at higher temporal frequencies. Experiment 2 revealed that stimuli within a transitional frequency range of 5 to 10 Hz were more likely to be perceived as transparent. While we did not find evidence for angular separation differentially affecting reported transparency, previous studies have demonstrated that feature binding appears to be reliant on surface segregation (Clifford, Spehar, et al., 2004; Moradi & Shimojo, 2004; Vigano et al., 2014). This is consistent with the conjunction discrimination observed here, which was attained through a large angular separation, a high temporal frequency, or a combination of both. 

This transitional frequency range of 5 to 10 Hz was further emphasized by the distinct and mostly separate ranges of alternation frequencies supporting accurate conjunction discrimination in Experiments 3 and 4. Experiment 3 demonstrated that at alternation frequencies below 5 Hz, the presentation duration of a single feature pair is sufficient to allow binding of even spatially segregated features (Fujisaki & Nishida, 2010; Holcombe & Cavanagh, 2001). In Experiment 4, the temporally distributed color–orientation conjunction within the check display was impossible to discriminate on the basis of a single presentation. Only intermediate frequencies of 7.5 to 15 Hz supported accurate conjunction discrimination. At low temporal alternation frequencies the integration of the feature information is impeded, whereas at very rapid frequencies (30 Hz) the checks fuse perceptually into a static plaid in the same way as the gratings, and individual colors and orientations cannot be resolved. 

To account for accurate feature binding here, specifically within the frequency range of 7.5 to 15 Hz, the correct color–orientation conjunction must be correctly decoded from activity early within the visual system (Figure 5a). Previous work has demonstrated the modulatory role of feedback signals in the early visual cortex (Andolina, Jones, Wang, & Sillito, 2007; Juan & Walsh, 2003; Lamme, Super, & Spekreijse, 1998; Shipp et al., 2009), which may be involved in conjunction discrimination in stimuli presented at rapid alternation frequencies. In this model (Figure 5b), feedback from higher areas targets double-duty neurons that code conjointly for orientation and color (Burkhalter & Van Essen, 1986; Gegenfurtner, 2003; Johnson et al., 2008; Leventhal et al., 1995). Feedback selectively targets and thus enhances the responses of those double-duty neurons responsive to one of the two orientations present in the display, resulting in a corresponding increase in the response to the associated color. In this way, targeted feedback allows the correct pairing of color and orientation to be decoded from the response profile of the population of double-duty neurons. Such a process, by means of perceptual transparency, can support synchronous perception of feature conjunctions (Clifford, Spehar, et al., 2004), effectively removing the perceptual asynchrony associated with color–orientation stimuli (Clifford et al., 2003; Moutoussis & Zeki, 1997). 

Figure 5

 

Schematic of a proposed conjunction identification mechanism. (a, b) The x- and y-axes represent the response of color- and orientation-selective neurons, respectively, to the stimulus attributes used in this experiment. Response profiles for each visual feature are centered on the colors (orange and blue) and orientations (±45° tilted from vertical) that are physically present in the stimulus. Each Gaussian blob represents the combined response to a presented feature pairing. (a) The neural response creates a binding problem when viewing a display like that in Experiments 1 and 4 because the same populations of neurons would be active regardless of the current color–orientation conjunction. Thus, the correct color–orientation combination (here, orange grating tilted rightward with a blue grating tilted leftward) is unable to be distinguished from the opposite combination from just the low level neural response. (b) To resolve this, feedback from higher areas enables the selection of a particular orientation (in this example, right-tilted gratings). This in turn boosts the signal of the neurons that jointly code for the preferred orientation and color, resulting in an asymmetrical neural response that reveals the correct color–orientation pair.

Taking into account what is known of the underlying physiology (Schiller, 1992; Smith, Bair, & Movshon, 2002) and supported by perceptual studies (e.g., Badcock, Clifford, & Khuu, 2005; Chubb & Nam, 2000), a simple rectification mechanism can serve to make the color–orientation pairing explicit under this framework. Half-wave rectification occurring early in the visual system (Smith et al., 2002) would yield oriented structure (Figure 6) if the light and dark color portions of the selected orientation are processed by separate on and off channels (Schiller, 1992). There is evidence to suggest that these on and off channels play a role in the perception of transparency (Bartley, 1939; Holcombe, 2001), although Holcombe (2001) pointed out that some equiluminant displays (i.e., those that cannot be separated by on and off processing channels) can also lead to the perception of transparency. Therefore, other processing channels that are responsive to equiluminant colors may be required in order to account for the observations of Holcombe (2001). 

Figure 6

 

Half-wave rectification of grating and check stimuli. Representation of how an early half-wave rectification mechanism may parse the experimental stimuli. In order to discriminate the color–orientation pairing in both types of stimuli, gratings and checks are first half-wave rectified into separate on and off channels, preserving the chromatic information. Note that in this figure, only the light strips are represented for clarity. Once separated, if the stimulus facilitates temporal integration, the combination of strips is used to arrive at the correct feature combination.

In Experiment 1, the effects of angular separation on conjunction discrimination could be explained through a combination of apparent motion and mutual inhibition. A lower angular separation would result in apparent motion between the gratings (Holcombe & Cavanagh, 2001; Kawabe & Miura, 2004; Nothdurft, 1991; Watanabe & Cavanagh, 1996), facilitating integration between—rather than within—gratings. At a low angular separation, the perception of a single, moving grating would be analogous to previous color-motion displays, which found that at these same frequencies conjunction discrimination was poor (Moradi & Shimojo, 2004; Vigano et al., 2014). However, mutual inhibition could also play a role here. Lower angular differences are more likely to activate populations of neurons with overlapping tuning curves (e.g., Bredfeldt & Ringach, 2002; Ringach, Shapley, & Hawken, 2002). If the orientations are too close (i.e., angular separations of 15° or less), lateral inhibition between mechanisms selective for nearby orientations would interfere with both selection from feedback and temporal integration processes (Blakemore, Carpenter, & Georgeson, 1970; Blakemore & Tobin, 1972), reducing conjunction discrimination. 

In Experiment 4, both grating and check displays can be separated by luminance in the same manner, yielding separate light and dark colored bars (Figure 6). Temporal integration within either (or both) of these pathways could then provide a coherent color signal from which the associated orientation could be extracted. However, if the alternation frequency is too high, temporal integration within mechanisms that are unselective for orientation can occur prior to half-wave rectification. This would result in an alternate perceptual interpretation of the stimulus as a static gray plaid rather than a pair of transparent gratings. However, neurons jointly selective for multiple features may still be engaged, based on results from studies that report adaptation to feature conjunctions that were either invisible (Blaser et al., 2005; Vul & MacLeod, 2006) or unattended (Houck & Hoffman, 1986; Humphrey & Goodale, 1998). In these studies, feature conjunctions were masked by means of a high temporal alternation frequency. However, resultant but weak color-contingent aftereffects were still observed, suggesting that jointly selective neurons were still able to respond to the physical characteristics of the stimulus. 

The conclusions of this study with regard to persistent surface representations can be likened to the phenomenon of visual persistence; indeed, a complex relationship exists between visual persistence and temporal integration (Farrell, 1984). The nature of this visual persistence (Coltheart, 1980; Shioiri & Cavanagh, 1992) is still a matter for speculation, although Moradi and Shimojo (2004) suggested that it may be similar to a form of iconic memory trace (Neisser, 2014; Sperling, 1960). Persistence acts over a brief timescale (Coltheart, 1980; Dixon & Di Lollo, 1991; Hogben & Di Lollo, 1985; Mezrich, 1984; Shioiri & Cavanagh, 1992), with short stimulus durations (as is the case here at higher alternation frequencies) resulting in a prolonged persistent afterimage (Dixon & Di Lollo, 1991; Hogben & Di Lollo, 1985). The relationship between rapidly presented stimuli can be described by a model whereby sustained activation of stimuli generates persistence while nonoverlapping activity inhibits it (Groner, Bischof, & Di Lollo, 1988), matching with the finding here that both temporal alternation frequency and a large angular difference are necessary to support accurate conjunction discrimination. However, it is important to note that there is a distinction between physical persistence, in which a persistent image is indistinguishable from a physically present counterpart, and informational persistence, which is likened to a higher order, fading trace of visual data (Badcock & Lovegrove, 1981; Bowling & Lovegrove, 1981; Coltheart, 1980; Di Lollo, 1984). In these experiments, any effects of persistence on conjunction discrimination would be better accounted for as traces of visual information because the color–orientation conjunction would physically disappear in the sum of the gratings, interfering with conjunction discrimination (Figure 4c). 

Together, our psychophysical results reflect the complex temporal dynamics of surface segregation and feature binding processes. We propose that the binding process can operate on persistent transparent representations when such representations are available. Accurate color–orientation judgments at frequencies exceeding 5 Hz depend on the rapid formation of these surface representations. These persistent representations allow sufficient time for feature conjunctions to be identified and extracted, circumventing the relatively low temporal resolution of the binding process itself (Holcombe, 2009). Further work is warranted to uncover the mechanisms in the visual cortex responsible for these complex interactions, and we are optimistic that the novel check stimulus and paradigm introduced here can be adapted and applied to that end. 

Acknowledgments

This work was supported by Australian Research Council Future Fellowship Grant FT110100150 (to C. W. G. C.), National Health and Medical Research Council Grant APP1027258 (to C. W. G. C.), and the Australian Research Council Centre of Excellence in Vision Science. The authors thank Kiley Seymour, Matthew Patten, and the reviewers of this article for helpful thoughts and comments on the research. 

Commercial relationships: none. 

Corresponding author: Colin W. G. Clifford. 

Email: [e-mail]. 

Address: School of Psychology, UNSW Australia, Sydney, Australia.