Contour extracting networks in early extrastriate cortex

Abstract

Abstract

Abstract:

Abstract  Neurons in the visual cortex process a local region of visual space, but in order to adequately analyze natural images, neurons need to interact. The notion of an “association field” proposes that neurons interact to extract extended contours. Here, we identify the site and properties of contour integration mechanisms. We used functional magnetic resonance imaging (fMRI) and population receptive field (pRF) analyses. We devised pRF mapping stimuli consisting of contours. We isolated the contribution of contour integration mechanisms to the pRF by manipulating the contour content. This stimulus manipulation led to systematic changes in pRF size. Whereas a bank of Gabor filters quantitatively explains pRF size changes in V1, only V2/V3 pRF sizes match the predictions of the association field. pRF size changes in later visual field maps, hV4, LO-1, and LO-2 do not follow either prediction and are probably driven by distinct classical receptive field properties or other extraclassical integration mechanisms. These pRF changes do not follow conventional fMRI signal strength measures. Therefore, analyses of pRF changes provide a novel computational neuroimaging approach to investigating neural interactions. We interpreted these results as evidence for neural interactions along co-oriented, cocircular receptive fields in the early extrastriate visual cortex (V2/V3), consistent with the notion of a contour association field.

Introduction

The visual system comprises a number of separate areas. These areas contain neurons that respond only to stimuli falling within a local extent of visual space, the classical receptive field (RF). The RF structure of neurons changes, and in particular enlarges, from the striate to the extrastriate cortex (Hubel & Wiesel, 1977; Van Essen & Maunsell, 1983). Human functional magnetic resonance imaging (fMRI) can segregate the cortex into regions that contain separate maps of the visual field (DeYoe et al., 1996; Engel, Glover, & Wandell, 1997; Sereno et al., 1995). Recently, fMRI neural model-based approaches have estimated population receptive field (pRF) sizes in different visual field maps (Dumoulin & Wandell, 2008; Kay, Naselaris, Prenger, & Gallant, 2008). These pRF sizes in degrees of visual angle resemble those measured with neurophysiology, including systematic size changes across eccentricity and between visual field maps (Amano, Wandell, & Dumoulin, 2009; Dumoulin & Wandell, 2008; Harvey & Dumoulin, 2011; Winawer, Horiguchi, Sayres, Amano, & Wandell, 2010). 

Knowledge of the neural RF properties is only a first step toward specifying the kinds of intercellular interactions required to adequately analyze the rich information content of natural images. Natural images are rich in contour information with cocircular or smooth-curvature structure (Elder & Goldberg, 2002; Geisler, Perry, Super, & Gallogly, 2001; Sigman, Cecchi, Gilbert, & Magnasco, 2001) that could be exploited by intercellular interactions beyond the classical RF (i.e., extraclassical RF interactions). Psychophysical methods devised to characterize visual sensitivity for detecting contours have led to the notion of an association field (Field, Hayes, & Hess, 1993). The association field specifies rules by which the outputs of orientation-sensitive neurons are combined into a network to optimally exploit the statistical regularities in natural images—in particular, cocircularity. This notion follows the Gestalt law of “good continuation” (Wertheimer, 1923). 

Here we ask whether we can expose the neural circuitry operating along the lines of the posited “association field” (Figure 1). This question can benefit from an approach that not only averages over large numbers of cells but also reflects (subthreshold) local neural circuitry operations—aspects to which fMRI is well suited (Logothetis & Wandell, 2004). Furthermore, pRF measurements consist not only of the classical RFs but also of extraclassical RF properties such as contour integration mechanisms (Dumoulin & Wandell, 2008; Smith, Singh, Williams, & Greenlee, 2001), providing an opportunity to separate these components using fMRI. 

Figure 1

 

The contour association field. The figure indicates the classical RF and extraclassical RF interactions along co-oriented, coaxial and co-oriented, cocircular aligned RFs proposed to process straight and curved contours, respectively. Drawn after Field et al. (1993).

We also take advantage of the way the conventional pRF mapping design (Dumoulin & Wandell, 2008) measures pRFs using bar-shaped apertures that move across the visual field in a direction perpendicular to the long axis of the bar. Because the passing bar stimulates a line of points across the visual field at the same time, examining the time course of responses only constrains the pRF measurement in the direction of bar movement and is insensitive to where along the bar the pRF lies. As such, several bar passes, from different directions, are needed to constrain the two-dimensional reconstruction of the pRF. Because the direction of bar movement is the same as the direction of pRF measurement, we can examine effects caused by processing of the stimulus content along the bar's edges. The way these edges are processed by the neural population will affect the pRF estimates. As such, the direction of the postulated extraclassical RF effects relative to the direction of pRF measurement by each bar pass is very important. Here, we manipulate the orientation of contours relative to this direction of pRF measurement and examine how this affects pRF size estimates. 

We used stimuli consisting of sparse contours known to elicit robust responses in the visual cortex (Dumoulin, Dakin, & Hess, 2008). While measuring from identical cortical locations, we carefully manipulated the contour properties within the pRF mapping stimuli. Because we measured from the same cortical location, nuisance factors that contribute to the pRF, such as hemodynamic response properties and spatial extent of our recording site, were constant across stimulus conditions. Therefore, we can safely assume that the changes in the pRF measured with different contour properties at the same cortical site are due to differences in neuronal contributions to the pRF. The sparse contours end either parallel or orthogonal to the measurement direction. When the contours end in the measurement direction, we hypothesized that the pRF sizes will increase due to the contribution of contour integration mechanisms. To verify that classical RF mechanisms cannot explain our results, we also processed the same stimuli using Gabor filters, a widely accepted model of V1 RFs. In summary, we propose that when we change the contour properties, we alter contour integration dynamics, which would result in different pRF sizes. Based on changes in pRF properties, we found evidence for a mechanism linking neurons with co-oriented, cocircular aligned RFs in V2 and V3 but not V1. 

Methods

Subjects

Measurements were obtained from six subjects (one female; aged 24–37); four subjects were naïve to the purpose of the study. All subjects had normal or corrected-to-normal visual acuity. All studies were performed with the informed written consent of the subjects and were approved by the Human Ethics Committee of the University Medical Center Utrecht in accordance with the World Medical Association's Declaration of Helsinki. 

Stimuli

The visual stimuli were generated in the Matlab programming environment (MathWorks, Natick, MA) using the PsychToolbox (Brainard, 1997; Pelli, 1997) on a MacBook Pro (Apple, Cupertino, CA). The subjects viewed the display through an angled mirror. The display configuration was back projected from outside the scanner room through the conductive glass on the front side onto a screen at the foot of the scanner. 

Bar-shaped apertures revealed the stimulus carrier (static contours; see Figures 2a and 3). The bars moved in discrete steps in eight different directions across the visual field uniformly sampled between 0° and 360°. To avoid broadband spatial frequency components introduced by hard aperture edges, edges on all sides were faded into the background following a 1/3°-wide raised cosine function. The bar width was 1° including faded edges on both sides. The total stimulus area had a diameter of 6°. The bar moved in sync with the fMRI data acquisition (a 0.3° step every 1.5 s). The bars were replaced by mean-luminance blocks every 48 s for 12 s (four mean luminance blocks). The insertion of mean-luminance blocks—a proper baseline—is essential for estimating the exact pRF sizes (Amano et al., 2009; Dumoulin & Wandell, 2008). A small red fixation dot was presented in the center of the stimulus area. 

Figure 2

 

Examples of the stimulus creation for straight contours (a–e) and curved contours (f–j). (a) The bar aperture. Low-pass and orientation band-pass filtered noise (b) was binarized (c), and pixels away from the borders were set to mean luminance (d). The final image was masked by the bar aperture (e). To generate curved contours whose contours ended in the direction of the pRF measurement, we filtered the same noise pattern twice (f, g). These patterns were blended in the bar aperture's faded edge (h), and edges were extracted using the same process described above (i). The final image was masked by the bar aperture (j).

Figure 3

 

Stimulus creation and rationale. The stimuli were created by masking black–white contours with bar-shaped apertures (left panels; see also Figure 2). Stimulus examples with horizontal and vertical directions of pRF measurement (middle panels) demonstrate that contour content rotates with the direction of pRF measurement (white dashed arrows, not shown in actual stimulus). Top panels (a, b) show relatively straight contours, whereas bottom panels (c, d) show curved contours with systematic orientation content only at the aperture edges. Orientation was either in the direction of pRF measurement, such that contour integration mechanisms should affect neural responses just outside the bar aperture (b, d), or orthogonal to the direction of pRF measurement, such that contour integration should remain within the bar aperture (a, c). The straight contours conditions (a, b) differ from each other along several stimulus properties and should cause co-oriented, coaxial integration. The curved contours conditions (c, d) are identical to each other except for the contour orientations in the faded bar aperture edge and should cause co-oriented, cocircular contour integration. In the right panels, a hypothetical classical neural RF, contour association mechanism, and the net pRF profile are shown as a red Gabor, red dashed lines, and black lines, respectively; none were shown in the actual experiment (see Figure 1). The net pRF profiles may differ due to different classical or extraclassical RF contributions. Specifically, we hypothesized that the association field would enlarge the net pRF size when operating in the direction of pRF measurement (b, d; solid line pRF profiles) compared with orthogonal to the pRF measurement (a, c; dashed line pRF profiles). Example movies of these two bar sweeps are available in supplementary materials.

The stimulus carrier consisted of randomly generated contours (Figures 2 and 3) (Dumoulin et al., 2008). Each presentation consisted of a new, randomly generated contour carrier. The straight and curved contours were created from the zero crossings of filtered noise patterns. We generated the noise patterns by assigning each image pixel a random value drawn from a uniform distribution between black (zero) and white (one). To generate straight contours, we filtered this noise pattern using both a low-pass filter (cutoff at 3 cycles/deg) and an orientation band-pass filter (30° cutoff) (Figure 2b). The resulting image was binarized (Figure 2c), and pixels whose distance was further away than 1/6° from the black–white border (zero crossings) were set to mean luminance (gray) (Figure 2c). The width of the contours was 1/3° and the approximate average distance between the contours was 2/3°. This contour carrier was masked by the bar aperture (Figure 2a) to create the final image (Figure 2d). To generate curved contours we filtered the same noise pattern twice, once using the same filters used to generate the straight contours (Figure 2f) and once without the orientation band-pass filter (Figure 2g). These two noise patterns were merged together in the bar aperture's faded edge (Figure 2h). Again the region directly bordering the zero crossings was binarized (black–white), whereas the other regions were set to mean luminance (gray) (Figure 2i). This image was masked by the bar aperture (Figure 2a) to create the final image (Figure 2j). Essentially, this means that within the bar aperture the contours were curved without an orientation bias but in the faded edge of the bar aperture the contours were bent into a particular direction (see Figure 3 for example images). 

Different contour carriers were randomly generated. Three random static contour carriers were presented every 500 ms for 300 ms (except during the mean-luminance blocks) in sync with every fMRI data acquisition and each step of the bar aperture (every 1.5 s). Within the 300 ms neither the contour carrier nor the bar aperture changed. On rare occasions, approximately 5% of the time, the same contour carrier was presented twice in succession. The subjects' task was to detect these events—a one-back task—and respond using a button press, ensuring attention to the orientation content of the stimulus. The average (and standard deviation) detection rate and d' (sensitivity index) were about 55% (19%) and 2.3 (0.6), respectively, across all scan runs and subjects (n = 177). 

Gabor filter analysis

We processed 50 randomly generated images from each of the four conditions shown in Figure 2. For this analysis, all the images contained centrally located, vertical apertures. The images were filtered with even-symmetric Gabor filters, with orientations stepping from 0° to 170° in steps of 10°. The vertical Gabor kernels (i.e., RFs) were defined as  

 

where λ = 0.33°, σ_y = 0.2°, and σ_x = 0.1°, giving an aspect ratio of two. These values correspond to physiologically plausible ratios (Jones & Palmer, 1987). After filtering, the filter output magnitudes were summed across filter orientation and across the vertical dimension of the image to give an estimate of total cortical activity as a function of distance along the direction of pRF measurement. For each image, pRF size was estimated by fitting a Gaussian function to the plot of activity against distance along the direction of pRF measurement (horizontal), and the best-fitting size (i.e., standard deviation) was recorded. The mean and standard error of the pRF size were calculated for each condition. The standard error of the difference between two conditions was taken to be the Pythagorean sum of the standard errors on the two conditions. Similar results were obtained using Gabor filters with λ = 0.33°, 0.45°, 0.67°, 0.94°, and 1.33°; aspect ratios of 1, 2, 4, and 8; and σ_y/λ ratios of 0.2, 0.4, and 0.6 (all combinations of these values). Therefore, the results of the Gabor filter analyses are valid for a wide range of Gabor parameters. 

Magnetic resonance imaging

Magnetic resonance images were acquired with a 3T Philips Achieva scanner and an eight-channel head coil (Philips, Best, Netherlands). Foam padding minimized head motion. Functional magnetic resonance images were acquired using a single-shot echo planar imaging sequence with 24 slices oriented orthogonal to the Calcarine sulcus with no slice gap (TR/TE 1500/30 ms, flip angle 70°, voxel size 2.5 × 2.5 × 2.5 mm, 166 time frames, scan duration about four minutes). Between eight and 12 scans were performed in each session. Each subject participated in at least two fMRI sessions. In a separate session, high-resolution T1-weighted MRI images were acquired with the following parameters: repetition time/echo time/flip angle = 10/4.6 ms/8°, effective voxel size 0.79 × 0.80 × 0.80 mm. 

Processing of anatomical and functional images

The T1-weighted anatomical MRI data sets were resampled to a 1-mm³ isotropic resolution. Gray and white matter were segmented from the anatomical MRI using FMRIB Software Library (FSL, Smith et al., 2004) and hand edited to minimize segmentation errors (Teo, Sapiro, & Wandell, 1997). The cortical surface was reconstructed at the white–gray matter border and rendered as a smoothed three-dimensional surface (Wandell, Chial, & Backus, 2000). Head movement and motion artifacts between and within functional scans were measured and compensated for (Nestares & Heeger, 2000). Functional data were then averaged across scans. Functional data were aligned to anatomical scans (Nestares & Heeger, 2000) and interpolated to the anatomical segmentation. 

pRF model-based analysis

We used the pRF model-based method to estimate the population pRFs. The pRF is defined as the region of visual space that stimulates the recording site (Dumoulin & Wandell, 2008; Jancke, Erlhagen, Schoner, & Dinse, 2004; Victor, Purpura, Katz, & Mao, 1994). Details of the pRF analysis are described in previous studies; both the pRF and hemodynamic response function (HRF) parameters were estimated from the same mapping stimuli (Amano et al., 2009; Dumoulin & Wandell, 2008; Harvey & Dumoulin, 2011; Winawer et al., 2010). 

Briefly, the pRF is modeled as a circular symmetric two-dimensional Gaussian. We also modeled the pRF as a circular symmetric two-dimensional difference-of-Gaussians model (Zuiderbaan, Harvey, & Dumoulin, 2012). The results of the Gaussian and difference-of-Gaussians models were nearly identical, so we report only the results of the Gaussian model. The Gaussian model parameters are center position (x, y) and spread (σ). All parameters are defined in standard units of degrees of visual angle. Using the pRF model, we can predict the fMRI response by a convolution of the pRF model with the stimulus sequence and the HRF. The optimal pRF model parameters are estimated by minimizing the sum-of-squared differences between the predicted and measured fMRI time series. The model parameters are computed for each cortical location from the corresponding fMRI signals (Dumoulin & Wandell, 2008). 

After estimating the pRF parameters using a canonical HRF, we determined one set of optimal HRF parameters for the entire recorded volume. The HRF parameters were determined by minimizing the residual sum of squares between the predicted and observed blood-oxygen-level-dependent (BOLD) responses over the entire recorded cortex where the pRF model explained at least 10% of the variance in the data. During this procedure, pRF parameters were kept constant. Next, the pRF parameters were refined to optimize the pRF parameters given this HRF (Harvey & Dumoulin, 2011). 

In this study, we first estimated the pRF and HRF parameters based on the average of all stimulus conditions. This yielded an unbiased estimate of the pRF model parameters. The pRF position parameters (x, y) were converted to the more traditional polar angle and eccentricity maps (i.e., from Cartesian to polar coordinates). Based on these maps, the visual field maps were delineated for the region of interest (ROI) analysis (Wandell, Dumoulin, & Brewer, 2007). 

Next, the pRF size parameter (σ) was refined for each condition separately, while keeping the position parameters (x, y) and HRF parameters fixed. The pRF size parameter was refined using an optimization algorithm (Fletcher & Powell, 1963) initiated from the optimal pRF size derived for all conditions. The pRF size difference was computed by subtracting the relevant experimental conditions and averaging voxels within each visual field map. The 95% confidence intervals were estimated from the pRF size difference variance corrected for upsampling within each ROI. 

Computation of aligned fMRI responses relative to pRF centers

In a visual field map, the response of a recording site is a combination of the visual stimulus and the pRF position and size. In order to compute an average response profile across all recording sites in a visual field map (eccentricity range 1°–2°), we temporally aligned each bar sweep relative to the center of the pRF. Specifically, we defined the time point at which the bar sweep crossed the estimated center of the pRF as time = 0. We could then average all bar sweeps across all recording sites and obtain the mean response to a particular stimulus within a visual field map. We used only the first three diagonal bar sweeps because the cardinal directions are interrupted by the mean-luminance presentations and because the final diagonal sweep is at the end of the fMRI time series and does not contain the full hemodynamic response. These average responses highlight differences in the raw response profile comparable with the duty cycle measurements (Li, Dumoulin, Mansouri, & Hess, 2007; Smith et al., 2001; Tootell et al., 1997) but ignore timing (phase) of the response mean-luminance presentations and are still contaminated by the HRF and noise in pRF position estimates. Therefore, these measurements are similar to but not necessarily identical to pRF estimates. 

Results

Rationale of isolating classical and extraclassical neural RF properties

We distinguished two properties of the pRF mapping stimuli: the carrier and the aperture (Figure 3, left panels). The carrier consisted of randomly generated sparse binarized contours (Dumoulin et al., 2008). The bar-shaped aperture reveals the carrier. The pRF properties are calculated based on the stimulus energy within the aperture—in this case, contrast energy (Dumoulin & Wandell, 2008). Importantly, the recording site as well as the stimulus aperture and contrast energy within the aperture are identical across all stimulus manipulations. Bar aperture orientation was yoked to contour orientation to maintain the relative orientation between the bar aperture and contour carrier. Every contour carrier presentation was unique and randomly generated. In these experiments it is crucial to realize that the pRF method measures the pRF size orthogonal to the bar orientation; it is insensitive to pRF properties parallel to the bar orientation. Essentially, each bar sweep constrains a one-dimensional reconstruction of the pRF. Multiple bar sweeps at different orientations are required to reconstruct a two-dimensional image of the pRF. The yoked presentation ensures an isotropic distribution of any directional classical or extraclassical RF interactions within each stimulus condition. 

The pRF size ( Display Formula ) estimate depends on a combination of neural and nonneural components (for a review, see Dumoulin & Wandell, 2008; Smith et al., 2001). Their relationship can be described as  

 

where σ_nRF is the mean size of the neuronal RF, σ_pv is the position variance of the neuronal RFs, and k is a constant factor that captures the nonneural factors. At the fMRI resolution the neural RF contains both classical (σ_cRF) and extraclassical (σ_ecRF) RF interactions:  

 

where the underlying assumption is that both types of RF interactions influence the fMRI signals but no assumptions are made on the actual biological implementation of these interactions. 

Here we compare the pRF size estimates derived from different stimuli, say stimulus a and b:  

 

 

In the actual experiment the stimuli in Figure 2a and b and Figure 2c and d were compared. Because we measured with the same stimulus aperture and the same recording site and extent, we assumed that the effects of position variance (σ_pv) and nonneural factors (k) cancel out (combining Equations 2, 3, and 4):  

 

 

Thus, comparing two stimulus conditions allows estimates of the neural RF differences (σ_nRF(a−b)). We explicitly model the contribution of the classical RF (σ_cRF(a−b)) by Gabor filters (Equation 1). We hypothesize that when our modeling shows no difference in the classical RF contribution, any neural RF differences reflect differences in the extraclassical RF contribution (σ_ecRF(a−b)). In our specific case, the directionality of the contour association field predicts increases in pRF sizes when the contours end in the direction of pRF measurement as compared with the condition where the contours end parallel to the direction of pRF measurement (schematically indicated in the right panels of Figure 3). Unlike the prediction generated by the Gabor models, the prediction of the contour association field is qualitative, not quantitative. 

fMRI responses differ with contour orientation

Example BOLD fMRI time series and estimated pRF position and profiles are shown in Figure 4. Time courses of a single V2 recording site are shown (Figure 4a) with the corresponding pRF estimates (Figure 4b, c). There are consistent differences in the time course, as illustrated by the average time course of the diagonal bar sweeps (second, fourth, and sixth bar sweep of the eight bar sweeps evident in Figure 4a). We chose to average the first three diagonal bars because mean-luminance blocks and end-of-scan do not interrupt them. For this subject, the full-width-at-half-maximum (FWHM) of the average response is wider for perpendicular compared with parallel contour stimuli in V1 and V2 (p < 0.001; Figure 4d, e, j, k). This analysis is not identical to the pRF analysis and is in many ways impoverished (e.g., imperfect alignment, overlapping responses, different pRF sizes, no modeling of the HRF, and removal of the timing). All of these will decrease the effect. Therefore, the observed differences are smaller. Nevertheless, the average time courses illustrate a consistent response difference between the parallel and perpendicular stimuli conditions for one subject. 

Figure 4

 

Example fMRI data, pRF fits, and average fMRI response. Purple and green indicate responses elicited by viewing bar-shaped apertures with contours orthogonal and parallel to the direction of pRF measurement (the stimuli are illustrated in Figure 3a and b, respectively). (a) fMRI time series from a single example recording site from one subject's V2 (no spatial or temporal smoothing). The fMRI time series clearly reveals eight peaks corresponding to the eight sweeps of the bar apertures across the visual field. (b) pRF estimates that capture most of the variance in the fMRI time series. Circles indicate the pRF's full width at half maximum within the visual field (same color scheme). These pRF models explain about 70% of the variance in the overall time series (r²) indicated by solid lines in panel a. (c) One-dimensional cross-sections though pRF's vertical extent, similar to Figure 3. In the example cortical location, the pRF size was larger when the subjects viewed stimuli containing contours oriented in the direction of pRF measurement (green). (d–o) The average response aligned relative to the center of the pRFs in a visual field map to the first three diagonal bars (shaded regions are 95% confidence intervals; subject S1). Top (d–i) and bottom (j–o) panels are the average responses for relatively straight and curved contours, respectively. The average time courses illustrate a consistent response difference between the two stimuli conditions for one subject.

V1 pRF changes are explained by classical RF properties

To calculate the changes in pRF size, we subtracted pRF size (σ_pRF(a−b)) measurements where the contours were oriented orthogonal to the direction of pRF measurement (Figure 3b, d) from measurements where the contours were oriented in the direction of pRF measurement (Figure 3a, c) at each recording site. The average pRF size difference for the two types of stimulus for one example subject and all subjects is shown in Figure 5a and c, respectively. The consistency of the pRF estimates is illustrated in Figure 5b, where independent subsets of the data set of V2 in Figure 5a are analyzed separately, yielding similar results. Similar results were obtained for other visual field maps. The pRF size differences vary systematically between visual field maps. 

Figure 5

 

pRF size differences measured with fMRI and a computational model (Gabor filters). (a) pRF size differences measured with relatively straight and curved contours with oriented ends in one subject. The mean pRF size difference in degrees of visual angle and 95% confidence intervals are shown for the different visual field maps averaged across the six subjects. Positive values indicate larger pRF sizes when the contours end in the direction of pRF measurement (Figure 3b, d). (b) The average V2 differences of the data set in panel a for different independent subsets of the data (odd, even scans and scans on day 1 and day 2). (c) Average pRF size differences measured with relatively straight and curved contours with oriented ends for all subjects. (d) The simulated size difference and 95% confidence intervals when processing the same stimuli with Gabor filters. The simulation results are quantitatively similar to the fMRI results for V1 but not for V2 or V3, indicating that the results in V1 can be driven by single neuron RF properties, and it is not necessary to postulate contributions of contour integration mechanisms. The changes in V2 and V3 pRF sizes in the curved contour condition cannot be explained by the model, indicating a possible contribution of contour integration mechanisms. A reverse pattern is visible in later visual areas hV4, LO-1, and LO-2, which cannot be accounted for by either the quantitative predictions of the Gabor model or the qualitative predictions of the association field.

In the early visual cortex, and in particular V1, we find increased pRF sizes when the orientation of the contours is aligned with the direction of pRF measurement. We processed the same stimuli using a bank of V1-like Gabor filters (Equation 1) and measured the pRFs based on the filter responses (Figure 5d). The result of this simulation both quantitatively and qualitatively mimics the fMRI pRF results in V1. Thus, this simulation suggests that the pRF size changes measured with fMRI can be explained by the filtering properties of single neurons' classical RFs (σ_cRF(a−b)), and it is not necessary to postulate a more complex “association field” mechanism (σ_ecRF(a−b)) to explain the results of this experiment. 

In the simulation, the difference in pRF size between the orthogonal condition (Figure 3a) and the parallel condition (Figure 3b) occurred in part due to the elongated Gabor filters, an aspect that is also reported for neural RFs (Jones & Palmer, 1987). But the difference may also occur because the contour stimuli are more narrowband (i.e., more like a sine wave) in a direction perpendicular to the contours than in a direction along the contours. The directional differences in spatial frequencies can also cause the difference. Because of this, the pRF size difference in the modelling does not critically depend on the RFs being elongated, and we obtained qualitatively similar results with a Gabor aspect ratio of one. The similarity of the fMRI and simulation in V1 both validates the ability of the pRF data analysis method to measure neural RF properties as well as the appropriateness of the Gabor filter model for V1 neural properties. 

Extraclassical mechanism processing “curved” contours in V2 and V3

While the curved contour stimuli differ at the blurred edges of the bar aperture as a function of orientation, they are identical within the central part of the bar aperture (Figure 3c, d). This stands in contrast with the relatively straight contour stimuli, which differ across the entire bar aperture for the two orientations. Simulated pRF sizes using a Gabor filter analysis as before do not differ between the orthogonal and parallel conditions (Figure 5d). This was the case for the specific filter parameters shown in Figure 5b, but also for a wide range of Gabor parameters extending within and beyond the biologically plausible range (Jones & Palmer, 1987). These simulations suggest that a passive bank of Gabor filters is not able to account for differential pRF sizes with the curved contour stimuli. 

Therefore, any effects of contour orientation on the pRFs measured with these stimuli require either (1) more complex classical RF properties that are not captured by the Gabor filter model or (2) the contribution of extraclassical RF interactions such as the association field. Specifically, we hypothesized that if there is a mechanism operating along the association field, the contours ending orthogonal (Figure 3d) to the direction of pRF measurement would be less likely to stimulate association field mechanisms lying outside of the bar aperture than would contours ending parallel (Figure 3c) to the direction of pRF measurement. Only V2 and V3 follow the qualitative predictions of the association field mechanism, suggesting that, in these areas, extraclassical RF interactions contribute to the pRF size differences in line with contour integration along the association field. 

In later visual areas, and in particular LO-2, both effects disappear and eventually reverse. Negative results cannot be accounted for by either the quantitative predictions of the Gabor filter model or the qualitative predictions of contour integration mechanisms along the association field. These negative values may be related to the classical RF or filtering properties in later visual areas but could also indicate other extraclassical integration mechanisms, such as those orthogonal to the local edge orientation as it is known to occur in “ladders” (Bex, Simmers, & Dakin, 2001; May & Hess, 2007a, 2007b, 2008). 

Relative pRF changes highlight pRF changes in early visual areas

So far the pRF size differences were reported in degrees of visual angle, and it is known that the pRF size changes systematically as one proceeds up the visual hierarchy. To judge the effect sizes properly the normalized pRF size changes need to be taken into account. We normalized the pRF size changes by the mean pRF size of each recording site:  

 

where y is the relative pRF size difference and x₁ and x₂ are the absolute pRF sizes for the two conditions at a particular recording site. The normalized pRF size changes for the two experiments are shown in Figure 6. The normalization procedure decreases the large absolute pRF size changes in hV4, LO-1, and LO-2 (Figure 5a) and highlights the large relative pRF size changes in the early visual field maps. 

Figure 6

 

Normalized pRF size changes for the different visual field maps. The data are identical to Figure 5a, but the pRF sizes are normalized according the mean pRF sizes at each cortical location (Equation 5). The error bars reflect 95% confidence intervals. The normalization procedure de-emphasizes the large absolute pRF size changes in hV4, LO-1, and LO-2 and highlights the large relative pRF size changes in the early visual field maps.

pRF size differences are not directly related to fMRI signal strength differences

The fMRI data analyses presented here depend on changes in pRF sizes. In contrast, conventional fMRI data analyses focus on changes in signal strength (amplitude), with the ultimate statistics being a measure of the signal-to-noise ratio. Theoretically, pRF sizes and signal strength are unrelated (i.e., a certain amount of fMRI signal is needed to estimate pRF sizes), but at adequate signal strengths pRF size measurements should be independent of signal strength. All data presented here exceeded a variance explained threshold of 30%; the mean and median variances explained above this threshold were 64% and 66%, respectively. In practice they may be related, however, either because of the underlying neural properties or due to deviations in methodological assumptions such as spatiotemporal linearity. It is therefore instructive to analyze the relationship between pRF size differences and signal strength differences. We computed both the amplitude of the fMRI signal (Figure 7a) and the variance explained (Figure 7b) of the pRF model (Dumoulin & Wandell, 2008). 

Figure 7

 

Relationship between pRF size and signal strength changes. For the identified visual field maps, pRF sizes are plotted as a function of a measure of the fMRI signal strength difference: fMRI signal amplitude in (a) percentage BOLD and (b) variance explained. The same visual field maps over the two experiments are linked with solid lines. Error bars reflect the 95% confidence intervals. In certain visual field maps increases in pRF size are associated with increases in amplitude or variance explained (V1, V2, V3), whereas in other visual field maps an opposite trend is apparent (hV4, LO-1, LO-2). Consequently, there is no systematic dependency of pRF size differences with amplitude or variance explained across visual areas.

The relationship between pRF size difference and variance explained difference for the identified visual field maps and the two experiments is shown in Figure 7. There is no systematic relationship within and between visual field maps. In some visual field maps an increase in pRF sizes is associated with increases in percentage BOLD signal change and variance explained (V1, V2, V3), but in other visual field maps an opposite trend is apparent (hV4, LO-1, LO-2). These results suggest that the pRF size changes cannot be easily predicted from fMRI signal strength changes. 

A related question is whether eye movements may underlie these pRF size changes. In a previous study (Levin, Dumoulin, Winawer, Dougherty, & Wandell, 2010), we demonstrated that isotropic eye movements systematically increase pRF size measurements proportionally for all pRFs (e.g., for all visual field maps). Therefore, eye movements cannot explain changes in pRF sizes in some but not other visual field maps. Nevertheless, we measured eye movements using the same stimuli outside the MRI environment using the Eyelink II system (SR Research, Ottawa, Canada) (Figure 8). The average standard deviation (σ) of eye movements around fixation of three subjects during the bar presentations was 0.16°. The difference in spread of the eye movements (standard deviation and 95% confidence interval) for relatively straight and curved contours was 0.0038° (−0.068°, 0.073°) and 0.0148° (−0.003°, 0.032°), respectively. The spread of the eye movements was not significantly different between conditions (overlapping 95% confidence intervals). 

Figure 8

 

Eye movement recordings in three subjects. Vertical and horizontal eye position are shown every 50 ms during stimulus presentation of curved contours, with contours ending orthogonal (a–c) or parallel (d–f) to the bar orientation. The three columns represent three different subjects (S1: a, d; S2: b, e; S5: c, f). No differences were found between conditions.

Discussion

Our fMRI results are based on changes in pRF size within the same cortical location. Many factors influence the pRF size, some neural and some not (for reviews, see Dumoulin & Wandell, 2008; Smith et al., 2001). Nonneural factors include head and eye movements, optical defocus, and both temporal and spatial HRF. There are also differences in neural contributions to the pRF. These include position scatter of the individual RFs of the recorded neural population and both classical and extraclassical neural RF properties. When measuring in the same cortical location, many of these parameters—including the nonneural factors and neural factors such as the position scatter—are not likely to vary. Therefore, we propose that pRF size changes as a function of the individual neural classical RF properties or due to changes in neural circuitry interactions elicited by the stimulus manipulations. 

In V1, a passive bank of V1-like Gabor filters mimics the pRF size changes measured with fMRI. This is consistent with previous observations that much of the V1 response variance is captured by contrast energy (Boynton, Demb, Glover, & Heeger, 1999; Dumoulin et al., 2008; Mante & Carandini, 2005; Olman, Ugurbil, Schrater, & Kersten, 2004). On the other hand, the data are also consistent with linking of neurons with co-oriented, coaxial RFs. Several observations demonstrate co-oriented, coaxial linking in V1 (Bosking, Zhang, Schofield, & Fitzpatrick, 1997; Gilbert, Das, Ito, Kapadia, & Westheimer, 1996; Gilbert & Wiesel, 1979, 1989; Kapadia, Ito, Gilbert, & Westheimer, 1995; Li, Piech, & Gilbert, 2006; Malach, Amir, Harel, & Grinvald, 1993; McManus, Li, & Gilbert, 2011; Nelson & Frost, 1985; Park, Cha, & Lee, 2013; Polat, Mizobe, Pettet, Kasamatsu, & Norcia, 1998). In essence, we cannot exclude co-oriented, coaxial RF interactions, but these interactions are not required to explain our results in V1. Importantly, we found no evidence for co-oriented, cocircular RF interactions in V1 along the association field (Field et al., 1993). 

Unlike in V1, the neurons in V2 and V3 appear to be able to exploit co-oriented, cocircular alignment (i.e., curved contours). The pRF size changes in V2 and V3 elicited by viewing of curved contours are not mimicked by a passive bank of V1-like Gabor filters. This suggests an active neural circuitry linking co-oriented, cocircular RFs in line with the proposed association field (Field et al., 1993). This result is consistent with nonhuman primate neurophysiological observations that extrastriate neurons in V2 and later visual areas respond to angles and curvature (Anzai, Peng, & Van Essen, 2007; Hegde & Van Essen, 2000; Ito & Komatsu, 2004; Pasupathy & Connor, 1999). These results are also consistent with both human and macaque V2 results, suggesting that neurons here respond strongly to these statistical regularities while V1 neurons do not (Freeman, Ziemba, Heeger, Simoncelli, & Movshon, 2013). 

Previous human and macaque fMRI studies implicate both early and higher visual areas in the process of contour integration (Altmann, Bulthoff, & Kourtzi, 2003; Kourtzi & Huberle, 2005; Kourtzi, Tolias, Altmann, Augath, & Logothetis, 2003). But unlike these previous studies, we exclude the higher visual cortex—and possibly V1—from the process of contour integration along the association field. One important difference is that our contours never formed any closed shapes. Even simple concentric shapes, such as mediated by contour closure, may involve specialized mechanisms in the extrastriate cortex (Altmann, Deubelius, & Kourtzi, 2004; Dumoulin & Hess, 2007; Tanskanen, Saarinen, Parkkonen, & Hari, 2008). A second difference is that our study is based on changes in pRF properties rather than signal amplitudes. We speculate that the integration of contour structure occurs in early visual areas and is communicated to later visual areas, where it may be reflected in signal amplitudes but not necessarily in changes of the pRF. In other words, RF interactions in V2 and V3 will likely affect later visual areas. On the other hand, we do observe pRF size changes in the higher visual cortex (in particular LO-2) that are opposite to the changes seen in the early visual cortex. This may be related to the classical RF or filtering properties in the higher visual cortex. Alternatively, it could also indicate other extraclassical integration mechanisms and reflect a role in contour integration but distinct from the early visual cortex. This effect may be interpreted as an increase of the pRF orthogonal to the contour orientation. As such, it may reflect integration of elements orthogonal to the individual element orientation also called “ladders” (Bex et al., 2001; May & Hess, 2007a, 2007b, 2008). 

To some degree, it may be surprising that our approach reveals active neural interactions because the bar width and stimulus energy remain constant. For example, consider a neuron whose RF lies on the edge of the bar and within a contour ending. This neuron would communicate with neurons across the bar edge (illustrated in Figure 3). But, because no stimuli are presented outside the bar, neurons with RFs outside the bar window are not expected to respond. However, subthreshold neural contact would have been made, giving rise to fMRI signals that are invisible to conventional neurophysiological approaches measuring spiking activity. We therefore suspect that this approach exploits the fMRI signal's sensitivity to subthreshold activity (Logothetis, Pauls, Augath, Trinath, & Oeltermann, 2001; Logothetis & Wandell, 2004). 

Our analysis measures the pRF sizes and their differences quantitatively (i.e., in degrees of visual angle) (Dumoulin & Wandell, 2008). The Gabor filter analysis is based on reported neural RF properties (Jones & Palmer, 1987). Both types of analysis yield qualitatively and quantitatively similar values in V1. From a methodological point of view, these stimulations indicate that the pRF size changes we measure with fMRI fall within a biologically plausible range. But we are cautious about drawing strong quantitative links due to the many unknowns that relate the two measures. 

In summary, using an approach that combines computational and neuroimaging techniques, we report evidence for RF interactions in V2 and V3 along co-oriented, cocircular RFs consistent with the notion of a contour association field. 

Supplementary Materials

mp4 S1 

mp4 S2 

mp4 S3 

mp4 S4 

Acknowledgments

We would like to thank the subjects for their participation and patience. This work was supported by Netherlands Organization for Scientific Research (NWO) grant (433.09.223 to S. O. D. and F. W. Cornelissen), CIHR grant (MOP53346 to R. F. H.), EPSRC grant (KAM, EP/H033955/1 to Joshua Solomon), NWO Veni grant (451.09.030 to B. R.), and NWO Talent grant (406.11.197 to M. B. and F. A. J. Verstraten). 

Commercial relationships: none. 

Corresponding author: Serge O Dumoulin. 

Email: [e-mail]. 

Address: Department of Experimental Psychology, Utrecht University, Utrecht, Netherlands.