Distinct perceptual grouping pathways revealed by temporal carriers and envelopes

Abstract

S. E. Guttman, L. A. Gilroy, and R. Blake (2005) investigated whether observers could perform temporal grouping in multi-element displays where each local element was stochastically modulated over time along one of several potential dimensions—or “messenger types”—such as contrast, position, orientation, or spatial scale. Guttman et al.'s data revealed that grouping discards messenger type and therefore support a single-pathway model that groups elements with similar temporal waveforms. In the current study, we carried out three experiments in which temporal-grouping information resided either in the carrier, the envelope, or the combined carrier and envelope of each messenger's timecourse. Results revealed that grouping is highly specific for messenger type if carrier envelopes lack grouping information but largely messenger nonspecific if carrier envelopes contain grouping information. These imply that temporal grouping is mediated by several messenger-specific carrier pathways as well as by a messenger-nonspecific envelope pathways. Findings also challenge simple temporal-filtering accounts of perceptual grouping (E. H. Adelson & H. Farid, 1999).

Introduction

Human vision excels at grouping local stimulus elements according to their time-varying properties (Alais, Blake, & Lee, 1998; Forte, Hogben, & Ross, 1999; Guttman, Gilroy, & Blake, 2007; Kandil & Fahle, 2001, 2003, 2004; Kojima, 1998; Lee & Blake, 1999a, 1999c; Leonards & Singer, 1998; Leonards, Singer, & Fahle, 1996; Rogers-Ramachandran & Ramachandran, 1998; Sekuler & Bennett, 2001; Usher & Donnelly, 1998)—see Blake and Lee (2005) for a review. Figure 1 shows a prototypical example of a temporal figure-ground segregation paradigm in which the contrast of figure and ground elements (Figure 1A) is square-wave modulated over time, and figure and ground elements alternate either out-of-phase (Figure 1B) or in-phase (Figure 1C). The relative temporal phase between figure and ground elements constitutes the only signal relevant to figure-ground segregation, as discrimination between figure and ground elements is physically impossible if relative temporal phase is zero (Figure 1C). 

Figure 1

 

Classic temporal figure-ground segregation paradigm. (A) Dot elements are spatially arranged to form a figure and a ground. (B) The contrast of figure and ground elements alternates over time in relative counterphase. (C) The contrast of figure and ground elements alternates over time in relative phase. Figure and ground elements cannot be discriminated if their relative temporal phase is zero.

The nature of visual mechanisms mediating temporal grouping remains controversial in part because of methodological issues (Adelson & Farid, 1999; Beaudot, 2002; Dakin & Bex, 2002; Fahle & Koch, 1995; Farid, 2002; Farid & Adelson, 2001; Morgan & Castet, 2002). In particular, Lee and Blake (1999c) argued that prototypical temporal figure-ground stimuli contain undesirable single-frame cues that compromise measures of human temporal grouping—that is, inspection of a single frame in Figure 1B is sufficient to reveal the spatial outline of the figure without need to compare the contrast of figure and ground elements over time. In an attempt to remove single-frames cues, Lee and Blake (1999c) designed stimuli consisting of drifting elements that reversed direction at random timepoints but where a subset of elements composing the figure reversed direction synchronously. The advantage of such stimuli, Lee and Blake argued, is that single-frame contrast cues are absent by virtue of the fact that the contrast of all elements remains fixed over time. However, Adelson and Farid (1999) applied temporal filtering (lowpass and bandpass) to Lee and Blake's stimuli and demonstrated that, while the overall contrast of each element remains fixed, the distribution of contrast energy within each element fluctuates from one temporal-frequency band to another as a function of time. The implication is that, when viewed through a temporal filter, direction-changing stimuli are contrast-modulated over time in the same way as the problematic contrast-changing stimuli that Lee & Blake hoped to avoid in the first place. 

The notion that temporal grouping operates directly on the output of temporal filters is important because it makes the simple prediction that contrast-changing and direction-changing elements should group perceptually, provided that both types of change elicit equivalent responses from temporal filters. In a particularly relevant study, Guttman, Gilroy, and Blake (2005) tested whether participants could perform figure-ground segregation in multi-element displays where each element was temporally modulated along a single dimension—or “messenger type”—such as contrast, position, orientation, or spatial scale. Critically, stimuli were engineered such that segregation was physically possible only if observers could group across mismatched messenger types. For instance, in one condition, observers had to group contrast messengers with position messengers in order to correctly identify the orientation of a rectangular figure. Data revealed that observers performed well with matched and mismatched messengers, and the authors concluded that performance was mediated by a single temporal-grouping pathway that discards messenger type and “abstracts” the temporal signal common to all stimulus elements. 

Guttman et al.'s (2005) results are consistent with the idea that grouping between contrast-changing and position-changing elements is mediated by temporal filters that extract time-varying components common to both messenger types. However, psychophysical, physiological, and computational evidence (see below) shows that temporal processing in human vision operates along distinct carrier and envelope pathways. By definition, carrier pathways operate on the linear—also referred to as “first-order” or “Fourier”—components of a stimulus, and linear components can be retrieved by operations such as convolving the stimulus with a temporal filter. By comparison, envelope pathways are sensitive to nonlinear—also known as “second-order” or “non-Fourier”—components of the stimulus. Linear operations such as convolution are blind to nonlinear stimulus components, and nonlinear operations such as rectification or squaring are needed to reveal second-order stimulus properties (Bracewell, 1986). What, then, is the role of temporal carrier and envelope pathways in temporal-grouping tasks such as figure-ground segregation? 

Human temporal sensitivity to temporal carrier-type stimuli is lowpass, peaks around 4 Hz, extends to approximately 40 Hz (Kelly, 1961, 1979; Watson, Barlow, & Robson, 1983), and strong evidence shows that temporal vision, just like spatial vision, is mediated by a collection of lowpass and bandpass filters sensitive to different parts of the temporal-frequency spectrum (Fredericksen & Hess, 1998, 1999; Hammett & Smith, 1992; Hess & Plant, 1985; Hess & Snowden, 1992; Legge, 1978; Lehky, 1985; Mandler & Makous, 1984). By comparison, sensitivity to temporal envelope-type stimuli is lowpass, peaks between 1 and 4 Hz, and has an upper-frequency cutoff of approximately 10 Hz (Derrington & Cox, 1998; Gorea, Wardak, & Lorenzi, 2000; Smith & Ledgeway, 1998). Strong psychophysical evidence shows that carriers and envelopes are processed along distinct perceptual pathways (Hammett & Smith, 1994; Ledgeway & Smith, 1994; Scott-Samuel & Smith, 2000). On the physiological front, electrode recordings from mammalian visual cortex show cells selective for carrier and envelope components of drifting stimuli (De Valois, Yund, & Hepler, 1982; Movshon, 1975; Movshon & Lennie, 1979; Zhou & Baker, 1993), and fMRI studies report a similar dissociation between carrier and envelope pathways (Ashida, Lingnau, Wall, & Smith, 2007; Dumoulin, Baker, Hess, & Evans, 2003; Vaina, Cowey, & Kennedy, 1999). Computational models of human motion incorporate distinct carrier and envelope pathways reported in the psychophysical and physiological literature (Sperling, 1989; Wilson, Ferrera, & Yo, 1992). 

In addition to the functional segregation between carrier and envelope pathways, a key finding in the spatial-vision and motion literature is that carrier pathways are selective for carrier properties but that envelope pathways are nonselective for carrier properties. For instance, carrier pathways are selective for carrier spatial frequency and orientation (De Valois et al., 1982; Movshon & Blakemore, 1973; Phillips & Wilson, 1984) but envelope pathways are largely nonselective for carrier spatial frequency or orientation (Mareschal & Baker, 1998, 1999; Wilson et al., 1992). We are not aware of studies that have explored the relative selectivities of carrier and envelope pathways in the temporal domain, but the reported nonselectivity of envelope pathways in the spatial and motion domains suggests that a similar nonselectivity in the temporal domain could account for the ability of Guttman et al.'s (2005) observers to extract common temporal signals from—and ultimately group across—mismatched messenger types. 

Two competing hypotheses emerge from the discussion above, and the purpose of the current study is to test empirically which of the two hypotheses provides a better account of temporal grouping. The first hypothesis—henceforth referred to as the “carrier hypothesis”—is derived from Adelson and Farid's (1999) analysis of Lee and Blake's (1999c) stimuli. This hypothesis assumes that: A—temporal grouping relies exclusively on linear temporal filters, and B—carrier pathways are nonselective for messenger type, in line with the notion that temporal filters can extract common temporal signals equally well across matched and mismatched messenger types. The carrier hypothesis accounts for Guttman et al.'s (2005) results, but it predicts that temporal grouping is blind to envelope components. 

The second hypothesis—henceforth referred to as the “envelope hypothesis”—assumes that temporal grouping is mediated by carrier and envelope pathways. This hypothesis assumes that, as suggested by the spatial-vision and motion literature: A—carrier pathways are selective for messenger type, and B—that envelope pathways are not selective for messenger type. The envelope hypothesis therefore predicts that temporal grouping across mismatched messengers relies only on envelope pathways and that, in the absence of envelope information, temporal grouping is restricted to matched messengers. Figure 2 illustrates how the distinction between temporal carriers and envelopes of visual stimuli provides the empirical means to compare and test predictions from the carrier and envelope hypotheses. 

Figure 2

 

Three carrier types and their corresponding envelopes. (A) The timecourse of figure and ground elements (top/red and bottom/green) is dictated by sinusoidal carriers (thin lines) with corresponding envelopes (thick lines). Instantaneous values for carriers and envelopes are indicated by labels A ₁ to A ₄ at an arbitrary time point. The top and bottom carriers are phase-delayed by 90° with respect to each other for illustrative purposes. (B) Same as A, but for compound carriers composed of 8 f + 11 f sinewaves. The envelope of the compound carrier has a fundamental frequency of 3 f. (C) Same as B, but for carriers generated by a stochastic binary point process. The figure carrier is phase-delayed by 30° relative to the ground carrier's fundamental frequency.

Figure 2 illustrates three classes of carriers (and their corresponding envelopes) that determine the timecourse of an arbitrary messenger type such as contrast or position. Envelopes are computed as the square-rooted sum of the squared carrier and the squared 90° phase-shifted version of the carrier, also known as the carrier's Hilbert transform (Bracewell, 1986). In Figure 2A, changes in figure and ground elements (top and bottom) are driven by pure sinusoidal carriers (thin lines). In the case of pure sinusoids, grouping across elements is physically possible only by comparing carrier values (A₂ and A₄) because envelopes (thick lines) are flat and therefore provide no temporal information that could differentiate figure from ground. Therefore, under conditions illustrated in Figure 2A, the envelope hypothesis predicts that temporal grouping with sinusoidal carriers should be rather easy for matched messengers but comparatively difficult for mismatched messengers. Alternatively, the carrier hypothesis does not differentiate between messenger types and therefore predicts that grouping performance should be similar across matched and mismatched messenger conditions. 

In Figure 2B, pairs of sinusoids (8 f and 11 f) form a compound carrier with a modulated envelope whose fundamental frequency (3 f) is given by the frequency difference between the two sinusoidal components. As we explain in Experiment 2, the phase of the envelope is dissociable from the absolute phases of the carrier components, and this independence allowed us to test conditions under which comparing messenger envelopes (B ₁ and B ₃) is informative but comparing messenger carriers (B ₂ and B ₄) is not. Under conditions illustrated in Figure 2B, the envelope hypothesis predicts that grouping should be possible across mismatched messenger types because the envelopes of compound-sinewave carriers, unlike those of sinusoidal carriers, are not flat and therefore informative. By comparison, the carrier hypothesis predicts that observer performance should suffer under both matched and mismatched messenger conditions because carriers are non-informative and envelope information is inaccessible. 

Finally, Figure 2C shows carriers and corresponding envelopes generated by a binary stochastic point process. Carriers are similar to those in Guttman et al.'s (2005) study, and because carrier and envelope are both rich with information, the carrier and envelope hypotheses both predict robust grouping across matched and mismatched messenger types. The experiment suggested in Figure 2C is not meant to discriminate between the two hypotheses but is meant instead to verify that our envelope hypothesis predicts human grouping performance across a wide variety of temporal stimuli, including stochastic ones used in Guttman et al's study. 

The three conditions outlined in Figure 2 form the basis of the three experiments in this study. Together, these conditions enabled us to test how temporal grouping operates on carrier information alone, on envelope information alone, and on combined carrier and envelope information. In line with the envelope hypothesis, data revealed that grouping across mismatched messenger types is virtually impossible in the absence of envelope information but readily doable if envelopes are informative. These findings imply that perceptual grouping is mediated by several distinct messenger-specific carrier pathways as well as by a single messenger-nonspecific envelope pathway. 

Methods

Participants

Participants HF, JO, and SH were two females and one male (21, 23, and 22 years old) with either normal or corrected-to-normal vision. Participants gave informed written consent were experienced psychophysical observers. The study complied with NIH standards of Human Participation Protection and was approved by North Dakota State University's ethical review board. All observers were naïve to the purpose of the study. 

Hardware, software, and calibration

Experiments were carried out on two identical Macintosh Dual 2.7 GHz PowerPC G5 workstations each equipped with an ATI Radeon X800 XT Mac Edition video card (10-bit DACs) driving a 22″ Iiyama HM204DT Vision Master 514 monitor (fast-decay phosphors) running at a 200 Hz frame rate and a spatial resolution of 800 × 600 pixels (we halved the effective resolution of the display to 400 × 300 using pixel doubling). Calibration measurements were made with a Minolta LS-110 handheld photometer, and the volt-to-luminance transfer function was linearized using the inverse of a best (least-squares) fit of a gamma function to the data. Averaged across the two monitors, minimum (black), and maximum luminance (white) after calibration were 0.20 and 100.05 cd/m ². Observers reported their observations with a Logitech RumblePad 2 gamepad. Experiments were scripted in Matlab 7.2 and used extensions from the PsychToolbox for OSX v. 3.x (Brainard, 1997; Pelli, 1997). 

Stimuli and procedure

The two messenger types used in this study consisted of changes in contrast and changes in position (i.e. spatial phase). In this section, we introduce a new stimulus type where modulations in contrast or position have identical temporal properties. Construction of these two messenger types is illustrated in Figure 3. 

Figure 3

 

Messenger types and timecourses. (A) Construction of contrast and position messengers. Messengers are composed of a dynamic test grating superimposed on a static pedestal grating of the same spatial frequency. Depending on the relative spatial phase ϕ _s between the test and the pedestal, messengers will oscillate either in contrast or in position. (B) Timecourses of test contrast for messengers belonging either to the figure (red) or ground (green). The two timecourses are phase-delayed with respect to each other by ϕ _t. Sinusoidal timecourses are shown for illustrative purposes, but timecourses can be replaced with other classes of waveforms (see Figure 2). (C) The dynamic test for two adjacent messengers is shown alone (left) or in conjunction with static pedestals that define position or contrast messengers.

In the spatial domain, each messenger consist of two superimposed sinusoidal gratings—a static pedestal and a dynamic test—windowed by a Gaussian aperture. As shown in Figure 3A, the pedestal and test are matched in spatial frequency but can differ in their relative spatial phase ϕ _s. While the pedestal remains static, the dynamic test changes in amplitude over time according to arbitrary functions such as those shown in Figure 2, and the dynamic test defines the messenger's timecourse (in this particular illustration, test amplitude varies sinusoidally). The relative spatial phase ϕ _s between the pedestal and the test determines whether the messenger's timecourse produces either oscillations in contrast ( ϕ _s = 0°) or oscillations in position ( ϕ _s = 90°). The fact that only spatial phase controls messenger type (contrast or position) is important because it implies that the test timecourse is identical for both messenger types and will therefore produce identical global power spectra. 

Figure 3B shows the sinusoidal timecourse of two messengers that belong either to the figure (red) or to the ground (green). The two messenger timecourses are phase-delayed from each other in time by an amount ϕ _t, and figure-ground discrimination should be impossible when the temporal-phase difference between figure and ground is minimal ( ϕ _t = 0°) and optimal when maximal ( ϕ _t = 90°). While one could argue, on technical grounds, that the maximal value of ϕ _t should be set to 180° rather than 90°, doing so introduces a confounding temporal symmetry in the stimulus (i.e., two sinewaves differing by 180° of phase are identical apart from a difference in sign). Figure 3C shows two messengers side by side and illustrates the dynamic tests alone (left) and in conjunction with their static pedestals (right). Position messengers shown in Figure 3 only approximate positional oscillations and retain undesirable contrast cues that we controlled for as described in 1. 

Figure 4A illustrates the spatial layout of the figure-ground discrimination task that we borrowed from Guttman et al. (2005). The display consists of a 6 × 6 array of messengers, and each messenger belongs to one of two populations (squares and circles) that can be assigned to one of two messenger types (contrast or position). The tasks is to report whether a 4 × 2 rectangle straddling the central 2 × 2 messengers is oriented vertically or horizontally. It is obvious from the symmetry of Figure 4A that the task is impossible with only knowledge of how the two messenger populations are spatially distributed. Figure 4B shows that a temporal-phase difference between figure (red) and ground (green) messengers is required to perform the task, and, critically, that the spatial distribution of messengers forces participants to integrate across the two messenger populations. 

Figure 4

 

Spatial stimulus layout and task. (A) Spatial layout of two populations of messengers (circles and squares) where each population can be assigned one of two messenger types (contrast or position). (B) Elements differ in the phase of their timecourses depending on whether they belong to the figure (red) or the ground (green). Observers report the perceived orientation (vertical or horizontal) of the rectangular figure and must necessarily integrate across messenger populations to perform the task.

Stimuli subtended 4.2 × 4.2 deg, and each messenger had a spatial frequency of 4.0 cpd and a Gaussian window with a spatial spread of σ = 0.125 deg. For each messenger, we randomized the absolute spatial phase common to the pedestal and test without affecting the relative spatial phase ( ϕ _s) that determines messenger type or the relative temporal phase ( ϕ _t) that differentiates figure from ground. This randomization removes instantaneous spatial-phase cues that could potentially contribute to performance and forces participants to rely exclusively on the timecourse of the dynamic test. Contrast ramped up and down smoothly over the full 0%-to-100% contrast range according to the Butterworth equation c( t) = 1/(1 + ( t/ t _c) ^{2 n}) where −0.5 s ≤ t ≤ 0.5 s, t _c = 0.15 s and defines the cutoff time at half height, and n = 4 and determines the steepness of the contrast variation. 

Data were collected in runs of 50 trials, and no fewer than 100 trials are included in every threshold reported herein. In all experiments, the relative temporal phase ϕ _t between figure and ground was varied adaptively in steps of 10° according to a two-up–one-down staircase routine. Psychometric data were fitted with two-parameter (translation and slope) cumulative-normal functions using a maximum-likelihood criterion. The fitted translation parameter corresponds to 75%-correct and was taken as our estimate of threshold ( ϕ _t ^75%) which we converted to a measure of sensitivity ψ as ψ = 90° − ϕ _t ^75%. In case where performance was too poor to reach threshold, we set ϕ _t ^75% to 90° and therefore set sensitivity to 0°. For each condition, we obtained confidence intervals (±2 SEM) using a bootstrap analysis (Efron & Tibshirani, 1993) of 100 iterations where each iteration resampled the data at each phase-delay level from a biased coin-flip process whose mean equaled the proportion of correct trials at that level. Data from each iteration were fitted in the same way as the original data, and we obtained a distribution of thresholds whose standard deviation we used to compute confidence intervals. 

Results

Experiment 1: Carrier-only information

In this experiment, the timecourse of the dynamic test consisted of a single sinusoid (see Figure 2A). Because the envelope of a sinewave lacks temporal landmarks (i.e., the envelope is flat and possesses no temporal-phase information of its own), observer performance in this experiment must be dictated exclusively by the messenger's temporal carrier. 

Each of the two messenger populations (circles and squares in Figure 4) can vary along one of two dimensions (contrast or position), and therefore a total of four (2 × 2) possible conditions can be assigned to populations 1 and 2 according to the following convention: contrast–contrast, contrast–position, position–contrast, and position–position. Examples of stimuli for the contrast–contrast, position–contrast, and position–position conditions are available as Quicktime movies in the supplementary video section linked to this paper. 

Figure 5 plots figure-ground performance (sensitivity) of two observers as a function of the dynamic test's temporal frequency for the four combinations of messenger types. Performance (sensitivity) is expressed as ψ = 90° − ϕ _t ^75% where ϕ _t ^75% corresponds to the threshold temporal-phase difference between figure and ground carriers. Performance in the position–position condition is high at low temporal frequencies and decays at higher frequencies. Performance in the contrast–contrast condition is lower overall but otherwise mirrors the position–position condition. Critically, however, sensitivity is essentially nil in the position–contrast and contrast–position conditions. Data clearly show that, for pure sinewave carriers with flat envelopes, observers simply cannot group across mismatched messengers but are quite capable to do so across matched messengers. 

Figure 5

 

Results from Experiment 1. Figure-ground segregation performance (sensitivity) for two observers (HF and JO) as a function of the temporal frequency of a pure temporal sinewave carrier. The two messenger populations are either matched in type: contrast–contrast (red) or position–position (yellow), or mismatched in type: contrast–position (green) and position–contrast (blue). Error bars show ±2 SEM.

To help determine whether grouping in the mismatched-messenger conditions (position–contrast and contrast–position) is categorically impossible rather than simply more difficult, one observer participated in a control experiment that replicated Experiment 1 with the exception that stimulus duration was increased by a factor of 8 ( t _c = 1.20 s rather than t _c = 0.15 s) while we held all other experiment parameters fixed. Results from this control experiment are reported in 2. Extending stimulus duration did not change the pattern of results shown in Figure 5, and this lends further support to the conclusion that temporal grouping for sinusoidal oscillations in contrast and position takes places along distinct carrier pathways with little or no pathway cross-talk. 

Experiment 2: Envelope-only information

In this experiment, the timecourse of each messenger was defined by two sinusoids with temporal frequencies f _A and f _B and amplitudes of 0.5. We freely manipulated the temporal frequency of f _A and derived f _B from the constraint f _B − f _A = 3.0 Hz. As shown in Figure 2B, the combination of two sinusoids with different frequencies produces an envelope that, unlike the flat envelope of Experiment 1 ( Figure 2A), is periodically modulated over its full depth range (0.0 to 1.0). The frequency difference between f _A and f _B components determines the envelope's periodicity, and while the envelope phase ϕ _Env is constrained by the relative phase of f _A and f _B ( ϕ _Env = ϕ _B − ϕ _A), it allows the components' absolute phases ( ϕ _A and ϕ _B) to covary. 

The last observation is crucial as it provides a way to statistically remove the messenger's carrier information without removing its envelope information. That is, for each messenger, we shifted each messenger's ϕ _A by a value independently and randomly sampled between 0° and 360° while holding the difference between ϕ _A and ϕ _B constant. This manipulation affords full control over the relative phase of ground and figure envelopes but destroys any systematic relationship between ground and figure carriers. Examples of stimuli for the contrast–contrast, position–contrast, and position–position conditions are available as Quicktime movies in the supplementary video section linked to this paper. 

Results from Experiment 2 are shown in Figure 6. Performance (sensitivity) is plotted as a function of the temporal frequency of f _A and is expressed as ψ = 90° − ϕ _Env ^75% where ϕ _Env ^75% corresponds to the threshold temporal-phase difference between figure and ground envelopes. To a first approximation, results for the matched-messenger conditions (i.e., contrast–contrast and position–position) are similar to those of Experiment 1, although recall that only envelope information—not carrier information—was available to observers in this experiment. Importantly, results for mismatched messengers (contrast–position and position–contrast) differ dramatically from those of Experiment 1 and show a definite performance peak between 10 and 20 Hz. 

Figure 6

 

Results from Experiment 2. Figure-ground segregation performance (sensitivity) for two observers (HF and JO) as a function of the temporal frequency of f _A (the lowest-frequency component of the sinewave pair). The frequency of the messenger's envelope was held fixed at 3.0 Hz throughout this experiment. The two messenger populations are either matched in type: contrast–contrast (red) or position–position (yellow), or mismatched in type: contrast–position (green) and position–contrast (blue). Error bars show ±2 SEM.

Why performance for mismatched messengers peaks at high carrier frequencies while performance is more broadly tuned for matched messengers is intriguing. A partial explanation could be that the envelope frequency (fixed at 3.0 Hz in the current experiment) is too close to the frequency of low-frequency carriers—a situation that is avoided in AM radio by constraining the carrier frequency to be orders of magnitude higher than the maximum envelope frequency to be transmitted (Rutledge, 1999). Such a constraint ensures that the envelope is sampled with a sufficient number of carrier cycles. However, given that carriers are physically uninformative in both the matched- and mismatched messenger conditions, the proximity of carrier and envelope frequencies cannot explain why performance remains high in matched-messenger conditions. Another possibility is that, taken at face value, data suggest that only high carrier frequencies are used by human vision to represent temporal envelope information. Testing this hypothesis would involve repeating Experiment 2 while prolonging overall stimulus duration, although this falls outside the scope of the present study. Indeed, the ultimate explanation for the high-frequency carrier tuning measured under mismatched-messenger conditions does not detract from the main conclusion of this experiment, namely that grouping across mismatched messenger types becomes possible under some conditions when the envelope of the messenger timecourse is informative. 

Experiment 3: Carrier and envelope information

In this experiment, the timecourse of each messenger consisted of a sparse all-or-none stochastic point process that consisted of exactly 10 binary changes throughout the stimulus duration (see Figure 2C). The timing of the 10 binary changes was randomly sampled from a uniform distribution covering the 0.0 s to 1.0 s temporal extent of the stimulus. The only restriction on the sampling was that two or more changes could not take place on the same frame. There were no other restrictions on the timing such as minimum or maximum inter-change intervals. The timecourses of figure and ground messengers were time delayed with respect to each other by variable amounts. In order to compare performance in all three experiments along similar metrics, we introduced time delays between figure and ground timecourses but recast those time delays in terms of degrees of phase of the timecourse's fundamental frequency. Given that stimuli were defined over a duration of 1.0 sec, messenger timecourses had fundamental temporal frequency of 1.0 Hz, and therefore each degree of phase of the fundamental frequency corresponds to 1.0 s/360°, or 2.78 ms/°. We defined sensitivity in Experiment 3 as ψ = 90° − ϕ _t ^75% where ϕ _t ^75% is the fundamental-frequency phase difference between figure and ground timecourses. Examples of stimuli for the contrast–contrast, position–contrast, and position–position conditions are available as Quicktime movies in the supplementary video section linked to this paper. 

Results from Experiment 3 are shown for two observers in Figure 7 where sensitivity for the two matched- and mismatched-messenger conditions is shown on the left axis and the corresponding time delay (in milliseconds) is shown on the right axis. Unlike Experiment 1, in which observers were largely incapable of grouping across mismatched messengers, results for stochastic timecourses show that performance for mismatched messengers is robust for both observers, especially HF. The measured time delays under each condition fall between 0 and ∼250 ms for observer HF and between 0 and ∼500 ms for observer JO. Such time delays are considerably longer than the ones measured in Guttman et al.'s (2007) study where the effects of synchrony and temporal structure on perceptual grouping were measured using stimuli similar, though not identical, to ours. First, Guttman et al. used 2.0 s rather than 1.0 s stimuli, and our stimuli were temporally windowed over a visible range of approximately 300 ms (t_c = 0.15). In addition, Guttman et al. imposed 15 changes per second whereas Experiment 3 from the current study imposed only 10 changes per second. The shorter duration of our stimuli combined with their lower density of changes per second likely accounts for the longer threshold time delays measured in the current experiment. 

Figure 7

 

Results from Experiment 3. Figure-ground segregation performance (sensitivity) of two observers (HF and JO) for sparse stochastic timecourses. Performance expressed in time delays (milliseconds) are shown on the right axis. The two messenger populations are either matched in type: contrast–contrast (red) or position–position (yellow), or mismatched in type: contrast–position (green) and position–contrast (blue). Error bars show ±2 SEM.

Discussion

The current study investigated human vision's ability to group local image elements according to their time-varying properties. In particular, we explored how figure-ground segregation depends on messenger type (changes either in element contrast or position) and on the carrier and envelope properties of messenger timecourses. Our purpose was to determine which of two hypotheses—the carrier hypothesis and the envelope hypothesis—better accounts for temporal-grouping performance in human vision. The carrier hypothesis assumes that temporal grouping: A—relies on linear temporal filters that do not discriminate between messenger types, and B—is blind to envelope information. By comparison, the envelope hypothesis assumes that temporal grouping: A—relies on carrier pathways selective for messenger type, and B—relies on envelope pathways nonselective for messenger type. 

The three studies reported herein measured grouping performance for matched vs. mismatched messengers in conditions where either the carrier alone ( Experiment 1), the envelope alone ( Experiment 2), or both the carrier and the envelope ( Experiment 3) contained grouping information. The carrier hypothesis predicts strong grouping performance for matched and mismatched messengers in Experiment 1 and poor performance for matched and mismatched messengers in Experiment 2. Alternatively, the envelope hypothesis predicts poor performance with mismatched messengers in Experiment 1 and robust performance for both matched and mismatched conditions in Experiment 2. Recall that Experiment 3 is not meant to discriminate between hypotheses but is intended to extend our findings to stochastic stimulus classes used in previous studies. 

Evidence from the current study clearly supports the envelope hypothesis: grouping for informative carriers with uninformative envelopes was restricted to matched messengers ( Experiment 1), and grouping for uninformative carriers with informative envelopes was possible both in matched- and mismatched-messenger conditions ( Experiment 2). As predicted by both hypotheses, grouping for informative carriers with informative envelopes was robust both in matched- and mismatched-messenger conditions ( Experiment 3). Results from these experiments suggest that temporal grouping relies on distinct carrier pathways that operate on specific messenger types as well as on an envelope pathway that discards messenger type and operates purely on temporal envelope structure. These findings mirror evidence from the spatial and motion literature that visual processing is selective for carrier properties such as spatial frequency or orientation (De Valois et al., 1982; Movshon & Blakemore, 1973; Phillips & Wilson, 1984) but that envelope processing is not (Mareschal & Baker, 1998, 1999; Wilson et al., 1992). The nonselectivity of envelope processing is a recurring theme in mammalian vision, and it appears to apply to temporal grouping as well. 

Single-frame cues are inevitable

A number of studies have made claims and counterclaims regarding the presence or absence of single-frame contrast cues in classic temporal-grouping stimuli (see Figure 1). In addition, several studies have gone to significant methodological lengths in attempts to remove these nominal transient contrast artifacts. Some of these attempts include: 1—modulating the orientation (Kandil & Fahle, 2001; Lee & Blake, 1999c) or direction (Lee & Blake, 1999c, 2001) of local stimulus elements rather than their contrast, 2—adding temporal contrast jitter to direction-reversing elements (Lee & Blake, 1999b, 2001), or 3—imposing zigzag rather than full direction reversals to drifting elements (Farid & Adelson, 2001). As we explain in this section and the next, such attempts to remove single-frame contrast cues are not only ill-fated but also unnecessary. 

Figure 8 shows local time-windowed power spectra for a counterphasing grating (panel A), a position messenger (panel B), and a contrast messenger (panel C). The horizontal and vertical axes of the spectra represent temporal frequency ( ω _t) and spatial frequency ( ω _s) respectively, and the DC level has been omitted for clarity. Local power spectra were obtained by windowing small temporal slices of the stimulus using a narrow sliding Gaussian aperture. Local power spectra for the counterphasing grating ( Figure 8A) show the expected waxing and waning as contrast fluctuates over time. Because the counterphasing grating is the only dynamic component present in position and contrast messengers, filters tuned to this counterphasing component could extract time-varying signals common to both messenger types and promote grouping. 

Figure 8

 

Local time-windowed power spectra for stimuli in current study panels show space–time power spectra for local time-windowed portions of (A) a counterphasing grating, (B) an oscillating position messenger, and (C) an oscillating contrast messenger. DC levels have been omitted in the spectra. Local time-windowed spectra were computed by selecting a narrow temporal slice of the stimulus (approximated by the dashed lines) with a sliding temporal Gaussian aperture.

Figure 8B shows the same local spectral analysis for an oscillating position messenger—unlike in panel A, local power in panel B remains constant but is redistributed over the spectrum as the messenger's position changes towards one direction or the other (notice how the orientation of the virtual line formed by the two spectral peaks changes as a function of time). Finally, Figure 8C shows a local spectral analysis for an oscillating contrast messenger where power predictably waxes and wanes as the messenger's contrast fluctuates over time. 

Figure 8B supports Lee and Blake's (1999c) point that direction-reversing messengers maintain a fixed overall contrast through time, but the figure also supports Adelson and Farid's (1999) point that while overall contrast is constant, the distribution of contrast energy (i.e., the spatiotemporal power spectrum) changes over time. From the perspective of a given filter tuned to a particular spatiotemporal frequency—in other words, a filter selective for a fixed region of the spectrum—contrast energy fluctuates in both the case where messenger direction changes (Figure 8B) or where messenger contrast changes (Figure 8C). The novel aspect that Figure 8 reveals is that, by definition, direction reversals necessarily involve a redistribution of spectral energy over time, and therefore attempts to prevent spectral fluctuations that define direction reversals are ill-fated. 

It should be clear from Figure 8B that modulating element orientation (Kandil & Fahle, 2001; Lee & Blake, 1999c) or spatial-scale (Guttman et al., 2005), just like a change in direction, also introduces a redistribution of contrast energy over time that filters with the appropriate spatiotemporal tuning can easily detect. Attempts to mask spectral fluctuations inherent to direction reversals by randomly jittering the contrast of elements (Lee & Blake, 1999b, 2001) are similarly futile because, as shown by Figure 8B, a direction reversal necessarily implies transient redistributions of spectral energy that filters with suitable tunings will respond to. 

The point that changes in direction, orientation, or spatial scale can always be reduced to a change in contrast through filtering is a critical but subtle one, as evidenced by Farid and Adelson's (2001) attempt to decouple changes in direction reversal from changes in contrast using zigzag rather than full direction reversals. Farid and Adelson's (2001) stimuli are necessarily prone to the same contrast-transient “artifacts” that the same authors highlighted in their earlier study (1999) because, by definition, direction reversals entail fluctuations in spectral energy such as those evident in Figure 8B. Critically, Figure 8 goes one step further than Adelson and Farid (1999) as it shows that filtering not only may reveal contrast transients inherent to changes in element direction, orientation, or spatial scale but also that filtering must reveal them. 

It is important to note that several studies have relied on computational analyses to reinforce claims that particular types of stimuli are devoid of contrast transients (Farid & Adelson, 2001; Guttman et al., 2005, 2007; Lee & Blake, 1999b). What these studies have in common, however, is that they commit their stimulus analysis to a single arbitrary temporal-frequency band that may or may not contain the relevant signal. As we have reported previously at conferences, a full-spectrum analysis of stimuli nominally purged of artifacts invariably reveals fluctuating frequency bands associated with direction reversals (Rainville, 2007a, 2007b), and this is unsurprising because changes in direction, orientation, or spatial scale necessarily involve transient redistributions of contrast energy somewhere in the spectrum. 

Single-frame cues are not artifacts

We have argued in the previous section that temporal-contrast cues—and the resulting fluctuations in filter outputs—are inevitable consequences of modulating element position, orientation, or spatial scale. But why are temporal-contrast cues considered as artifacts that should be eliminated from figure-ground stimuli in the first place? It is important to revisit this question in order for the study of temporal grouping to move forward. 

The figure-ground segregation paradigm shown in Figure 1 has evolved from perceptual-grouping studies in which stimulus features were static. For instance, Beck (1966, 1967) used stimuli in which figure and ground elements differed in their relative orientation and/or size. In an elegant paper, Bergen and Adelson (1988) showed that perceptual grouping with Beck-type stimuli correlates well with the differential response of a spatial filter applied to figure and ground elements. This simple spatial-filtering account of figure-ground segregation is no different in spirit than Adelson and Farid's (1999) temporal-filtering account of perceptual grouping with Lee and Blake's (1999c) original direction-reversing stimuli. 

Figure 1B shows a stimulus that alternates between figure and ground over time, and it is likely from such schematics that the flawed single-frame “artifact” argument is born. Critics of temporally alternating figure-ground displays argue that segregation has little to do with temporal mechanisms because a purely spatial analysis of a single frame is sufficient to reveal the outline of the figure or the ground. This argument is especially compelling if the rate of alternation between figure and ground is slow. However, the logic of this single-frame “artifact” argument is flawed because human vision is only capable of segregating figure from ground in Figure 1B because is it sensitive to events that take place at different moments in time. As shown in Figure 1C, no physical distinction can be made between figure and ground if both oscillate with the same temporal phase (or if alternations between figure and ground are averaged over long time periods). 

The “purely spatial” analysis that critics argue could take place on a single frame is only possible if some prior temporal preprocessing (i.e., temporal filtering) has taken place in order to isolate a particular frame from frames that precede it. Other than causality (which is of no importance to this specific argument), there are no categorical differences between filtering over space and filtering over time. Adelson and Farid's (1999) temporal-filtering argument is perfectly valid insofar as it shows that direction reversals involve a change in the temporal-frequency spectrum over time, but the distribution of contrast energy over time is no more artifactual than its distribution over space. 

The local spectral properties of contrast and position messengers

Why did the carrier hypothesis fail to predict grouping across mismatched messengers in the current study? That is, why did temporal filtering fail to reveal dynamic components (i.e., the counterphasing test gratings) common to both contrast and position messengers? 

Despite their identical global spectral properties, contrast and position messengers have significantly different local spectral properties. Although filters tuned exclusively to the counterphasing grating would produce identical responses to contrast or position oscillations ( Figure 8A), the combination of the dynamic test with the static pedestal leads to significantly different patterns of energy distribution over space and time ( Figures 8B and 8C). 

Early visual pathways differ in their space–time selectivity (De Valois & De Valois, 1982), and the dissimilarity between the local spectral properties of contrast and position messengers likely elicited responses from distinct pathways that do not feed into the same temporal-grouping mechanism. This high degree of pathway selectivity motivates further studies in which selectivity for other messenger dimensions such as orientation, spatial scale, disparity, speed or direction may also be manifest. Neurophysiological studies, including EEG and fMRI, should further elucidate the mechanisms that underlie the functional segregation of carrier and envelope pathways involved in temporal grouping. 

Supplementary Materials

Supplementary video - Beat contrast 

Supplementary video - Beat mixed 

Supplementary video - Beat position 

Supplementary video - Sine contrast 

Supplementary video - Sine mixed 

Supplementary video - Sine position 

Supplementary video - Stochastic contrast 

Supplementary video - Stochastic mixed 

Supplementary video - Stochastic position 

Appendix A

Controlling for contrast cues

The time-varying messengers used in the present study are ideal in the sense that changes in contrast and position were engineered to have identical timecourses and power spectra, and that they are therefore indiscriminable if one had to rely exclusively on temporal filters tuned to the dynamic test. However, although contrast and motion messengers have identical total-energy contents, an unfortunate property of the dynamic test + static pedestal technique illustrated in Figure 3 is that contrast messengers have a peak Michelson contrast of 100% whereas motion messengers peak only at 70.7%—a consequence of adding two spatial sinusoids in quadrature phase rather than in phase. A second but less problematic property of our motion messengers is a faint contrast pulsing owing to an imperfect approximation of changing motion. 

The dissimilarity in peak Michelson contrast could have the unwanted side-effect of promoting segregation between contrast and motion messengers. This potential confound cannot explain successful grouping across messenger types in Experiments 2 and 3, but it could perhaps account for failure to group across messenger types in Experiment 1. 

To test for this possibility, we repeated key conditions of Experiment 1 on observer HF using counterphasing Gabors as contrast messengers and sinusoidally drifting Gabors as motion messengers. While this manipulation sacrifices some of the advantages of our original stimuli, it has the upside that Michelson contrast is equated across contrast and motion messengers and remains constant for motion messengers. Results from this control experiment were virtually identical to those of HF in Experiment 1, and we conclude that unwanted contrast cues had no impact on our results. 

Appendix B

Extending duration of stimuli in Experiment 1

We replicated Experiment 1 with observers SH and extended stimulus duration eightfold ( t _c = 1.20 s rather than t _c = 0.15 s). Given that the shape of the Butterworth window scales with duration, we ensured that the slope of the contrast ramp-up and ramp-down matched that of the original stimulus in Experiment 1 by changing the window's order n from 4 to 32. We reduced the sampling of temporal-frequency conditions in the control experiment by a factor of 2, and we tested the matched-messenger (contrast–contrast and position–position) as well as the mismatched-messenger (contrast–position and position–contrast) conditions of Experiment 1 for original—as well as for extended-duration conditions. Results for observer SH are shown in Figure B1. 

Figure B1

 

Results for original and extended viewing duration in Experiment 1. Figure-ground segregation performance (sensitivity) for one observer (SH) as a function of the temporal frequency of a pure temporal sinewave carrier. The two messenger populations are either matched in type: contrast–contrast (red) or position–position (yellow), or mismatched in type: contrast–position (green) and position–contrast (blue). Results for the original viewing duration ( t _c = 0.15 s) are show on the left, and results for the extended viewing duration ( t _c = 1.20 s) are shown on the right. Error bars show ±2 SEM.

Results from the control experiment show that the pattern of performance observed in Experiment 1 does not change qualitatively from original- to extended-duration conditions. As with the original-duration condition of Experiment 1, the extended-duration condition of the control experiment shows that performance with sinusoidal carriers is high for matched-messenger types and nil for mismatched-messenger types. The superior performances in the position–position conditions over the contrast–contrast conditions, as well as the overall performance decay with temporal frequency, also mirror each other in the original- and extended-duration experiments. 

Prolonged viewing of mismatched-messengers with sinusoidal carriers does not enable figure-ground segregation. This result reinforces the conclusion that temporal grouping in the absence of envelope information is mediated by distinct messenger-specific pathways that exploit carrier information. 

Acknowledgments

This work was supported by COBRE grant P20 RR20151-02 from the National Institute of Health (NEI) to the Center for Visual Neuroscience at North Dakota State University (S.R.) and a North Dakota State University Presidential Fellowship (A.C.). 

Commercial relationships: none. 

Corresponding author: Stéphane Rainville. 

Email: [email protected]. 

Address: Center for Visual Neuroscience, Department of Psychology, North Dakota State University, Fargo, ND, 58105, USA.