Inconsistency between peri-saccadic mislocalization and compression: Evidence for separate “what” and “where” visual systems

Abstract

The view of two separate “what” and “where” visual systems is supported by compelling neurophysiological evidence. However, very little direct psychophysical evidence has been presented to suggest that the two functions can be separated in neurologically intact persons. Using a peri-saccadic perception paradigm in which bars of different lengths were flashed around saccade onset, we directly measured the perceived object size (a “what” attribute) and location (a “where” attribute). We found that the perceived object location shifted toward the saccade target to show strongly compressed localization, whereas the perceived object size was not compressed accordingly. This dissociation indicates that the perceived size is not determined by spatial localization of the object boundary, providing direct psychophysical evidence to support that “what” and “where” attributes of objects are indeed processed separately.

Introduction

Based on evidence from studies of electrophysiology, anatomy, and behaviors of patients with brain injuries, Ungerleider and Mishkin suggested a primate vision model comprised of relatively separate “what” and “where” systems that correspond to ventral and dorsal streams, respectively (Mishkin, Ungerleider, & Macko, 1983; Ungerleider & Mishkin, 1982). According to this view, the ventral stream is for object vision (what size, shape, and color of objects are), and the dorsal stream is for spatial vision (where objects are). While agreeing on the fundamental difference between the two streams, Goodale et al. argued that the distinction should be more in the output requirement, and they proposed “what” and “how” systems (Goodale & Milner, 1992; Goodale & Westwood, 2004). This revised model contends that the separation is characterized by the dorsal stream being involved in action control (e.g., grasping and reaching by hand to targets) and not specifically in spatial perception. Furthermore, it has been proposed that there may be two different spatial representations in the dorsal stream, one for spatial perception and another for action (Burr, Morrone, & Ross, 2001; Rizzolatti & Matelli, 2003). 

According to the two-system theories, functional error or impairment may occur in one stream but not another. There has been accumulating evidence from psychophysical studies showing that perceptual functions and motor functions can be dissociated (Churchland, Gardner, Chou, Priebe, & Lisberger, 2003; Dubrowski & Carnahan, 2002; Goodale, Pelisson, & Prablanc, 1986). However, there is very little psychophysical data to directly support the view of separate “what” and “where” systems. In this paper, we used the peri-saccadic perception paradigm to investigate whether error can occur in one system but not another. 

It is well known that stimuli flashed during saccades in lit conditions may be perceived as shifted toward the saccade target. In particular, items flashed between the previous fixation and the saccade target are mislocalized in the same direction as the saccade, whereas items flashed beyond the saccade target are mislocalized in the opposite direction. Almost all previous studies have used thin stimuli (usually less than 1° wide) to probe across a wide visual space (up to 40° wide) and found consistent evidence that supports compressed peri-saccadic localization (Honda, 1993; Lappe, Awater, & Krekelberg, 2000; Ross, Morrone, & Burr, 1997), which further led to a concept of compressed space. To understand if the observed pattern of mislocalization implies a corresponding perceptual compression of object size, a few studies have directly investigated the perceived size of objects centered at the saccade landing point, where compression is supposed to be the largest (Matsumiya & Uchikawa, 2001; Noritake et al., 2009; Sogo & Osaka, 2005). The conclusions are somewhat inconsistent, which may be due to different paradigms used, but the general finding is that the overall width of a group of objects may be compressed depending on configuration, whereas solid objects are less distorted or not at all. None of these studies investigated mislocalization and size compression in the same condition, and therefore, the paradigms could not allow investigation of the association between size perception and visual localization. In the present study, we investigated whether peri-saccadic localization error is consistent with size distortion across a wide range in visual space, which we believe will help understand the dissociation between the “what” and “where” systems from a psychophysical perspective. 

Methods

Experiment setup

Our research followed the tenets of the Declaration of Helsinki and was approved by the institutional review boards at the University of Massachusetts Boston and the Schepens Eye Research Institute. 

Two authors (GL and TG) and two naïve subjects (ET and RP) participated in the study. Stimuli were presented on a 21-inch Dell P1130 monitor with a resolution of 1024 × 768 pixels and a refresh rate of 100 Hz. The monitor spanned 44° × 33° at the observation distance of 36 cm. Eye movements were recorded using an EyeLink II eye-tracker system (SR Research, Ottawa, Canada), which samples at 500 Hz and has an average accuracy of 0.5°. All experiments were performed in a normally lit room (688 lux). Screen background was black (6.5 cd/m²), and stimuli included a fixation marker (1° crosshair, red, 33 cd/m²), a saccade target (1° round dot, red, 33 cd/m²), and a flashed bar (1°, 4°, or 8° wide and 0.5° high, white, 120 cd/m²). 

Each trial started with fixation at 10° left of screen center (Figure 1). After a random delay (1000 to 2000 ms), the fixation cross disappeared and the saccade target appeared at 10° right of screen center for 600 ms, at which point observers made a saccade to the saccade target. Sometime between 100 ms before and 100 ms after the anticipated saccade onset, a horizontal bar was flashed for one frame centered at one of six possible locations along the horizontal midline (−10, −5, 0, 5, 10, and 15° relative to the horizontal screen center). Following their saccades, participants reported the perceived location of the bar by using a standard PC mouse to click at the two end points of the perceived bar. The cursor was only visible during the report phase. 

Figure 1

 

Presentation of stimuli. When subjects made saccades from −10° to 10°, bars in different lengths (1, 4, 8°) were flashed at different locations (−10, −5, 0, 5, 10, 15°). Subjects used a mouse cursor to indicate the perceived locations of the two end points of the flashed bar.

A baseline test was also performed in which the identical procedure was followed, except that subjects instead fixated on the cross at −10° during the presentation of the flashed bar. 

Mislocalization modeling

To investigate if localization is associated with size perception, our approach was to first use the reported localization of bar end points to estimate what the perceived length of the bars should be if localization (spatial vision) and size perception (object vision) were perfectly consistent. This estimate was then compared with the reported bar length. For this purpose, we used a saccadic mislocalization regression model consisting of a Gaussian curve and an error function curve (integral of Gaussian): 

$\mathrm{P}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{e}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{d}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=A·\exp (-\frac{(t-\mu {)}^2}{2{\sigma }^2})+B·erf\left(\frac{(t-\mu {)}^2}{2{\sigma }^2}\right),$

(1)

where σ represents the width of the mislocalization range, μ represents the time of the mislocalization peak, and t is the bar onset time relative to saccade onset. The inclusion of the error function is based on our observation of a systematic difference between the perceived locations long before and long after saccades. This difference also occurred in many studies by other researchers (Awater & Lappe, 2004; Lappe et al., 2000; Ross et al., 1997). We think that this effect is due to a systematic visual localization bias (under- or overestimation) during fixations. Because this error varies with eccentricity, it may change after the gaze point moves by 20°. Therefore, an error function component is added to model the error change. 

In our model fitting, amplitudes of the Gaussian and error function components (A and B) are variable with veridical locations, and we used a 2nd order polynomial curve model for each of them. Only the data for the end points of the 1 and 4° long bars were used for mislocalization modeling (Figure 2), because short stimuli approximate sampling points more closely than long stimuli do. Furthermore, excluding the 8° long bars from modeling enabled the prediction for the 8° bar trials solely based on data from other (1 and 4°) trials, which is preferable for the evaluation of the model. The end points of the 1 and 4° long bars that formed the basis for the model were distributed across 24 different veridical locations from −12 to 17°. 

Figure 2

 

Saccadic mislocalization of horizontal bar end points (moving average). For mislocalization modeling, the end points of the 1 and 4° long bars were used, which were located at 24 different veridical positions.

Results

Figure 3 shows the perceived locations of the bars (presented as bar centers) versus bar onset time relative to saccade onset for all four subjects, as well as their mislocalization model based on end points. As can be seen, the mislocalization pattern for the horizontal bars is identical to the typical pattern seen in previous studies using vertical bars (Lappe et al., 2000; Ross et al., 1997). 

Figure 3

 

Left column: The mislocalization pattern of horizontal bar centers that was observed in all subjects in this experiment was very similar to that in previous experiments using vertical bars. Veridical locations of bar centers were at −10, −5, 0, 5, 10, and 15°. Curves are moving averages. Right column: Mislocalization regression model (based on the data for end points of 1 and 4° long bars) versus veridical bar location and bar onset time. The gray semi-transparent plane indicates zero error. These distributions were used to predict perceived bar length.

One of the previous findings that was thought to support the concept of space compression was that the distance between multiple flashed vertical bars is perceived as reduced during saccades (Ross et al., 1997), and we also replicated that result (not reported here). To follow the same reasoning, we used the localization error model to predict the perceived length of flashed horizontal bars by first interpolating the perceived locations of the two end points separately based on the mislocalization model and then calculating the distance between the two predicted points. Figure 4 shows the bar length reported using the mouse cursor versus the predicted bar length, for those bars presented within 50 ms before and after saccade onset. For each bar length, the linear relationships between reports and predictions are illustrated in the figure. If the prediction were correct, the linear slope would ideally be 1. However, the slope was much less than 1 (−0.08 to 0.33), and the linear relationship was weak (all R ² were within the range 0.0–0.16 with one outlier: R ² = 0.3 for 8° long bars in subject RP). 

Figure 4

 

Reported bar length versus predicted bar length based on the mislocalization model. Linear relationships were examined separately for bar lengths 1° (blue triangle), 4° (red star), and 8° (black circle). The slope was much less than 1 (−0.08 to 0.33), and the linear relationship was weak (all R ² were within the range 0.0–0.16 with one outlier: R ² = 0.3 for 8° long bars in subject RP).

Furthermore, we examined reported bar length as a function of absolute mislocalization error and as a function of absolute bar onset time (defined here as the absolute time difference between bar onset and saccade onset). The purpose of this analysis was to investigate whether larger localization error is associated with shorter perceived length (smaller bar onset time corresponds to larger localization errors). Figure 5 shows representative results from subject TG, plotting the ratio of reported length and veridical length versus absolute bar center localization error or bar onset time. Linear regression was computed for each veridical bar length, and the flat lines may indicate that reported bar length was independent of localization error as well as bar onset time. 

Figure 5

 

Ratio of reported length and veridical length versus localization error (left), and absolute bar onset time relative to saccade onset (right). Flat regression lines may indicate that the reported bar length was independent of localization error as well as bar onset time.

Both numerically and visually, the slope of linear regression in Figure 5 depends upon the units of the horizontal and vertical axes. Therefore, to quantitatively examine whether mislocalization is associated with perceptual size change, two large mislocalization conditions are assumed: 10° localization error and 0 ms bar onset time, respectively, and the predicted bar length is calculated based on the type of linear regression shown in Figure 5. Table 1 lists predicted bar length in these strong mislocalization cases based on the regression for each subject. The results are very close to the reported lengths in the baseline condition (without eye movements) with differences not exceeding approximately 1°. 

Table 1

 

Estimation of perceived bar length (in degrees) based on linear regression of bar length versus localization error and bar onset time. Estimation is calculated for two strong would-be compressed conditions, assuming localization error is 10° and bar onset time is 0 ms, respectively. Bar lengths reported in fixation condition are also listed. Estimation in saccade conditions and report in fixation condition were similar.

	True size	ET	TG	GL	RP	Mean
Estimation according to localization error (10°)	1.0	1.10	0.94	0.76	1.09	0.97
	4.0	4.07	4.11	3.99	4.51	4.17
	8.0	6.00	6.99	6.47	5.81	6.32
Estimation according to onset time (0 ms)	1.0	1.11	0.96	0.78	1.09	0.98
	4.0	4.25	3.96	4.26	4.97	4.36
	8.0	6.55	6.64	6.92	6.53	6.66
Fixation condition	1.0	1.18	1.27	0.92	1.19	1.14
	4.0	4.16	4.61	4.74	4.85	4.59
	8.0	6.90	7.41	7.71	7.71	7.43

Discussion

In the physical world, it is legitimate to describe a space using discrete points, and it is true that the shape and size of an object in that space can be measured by locating some of those points. Non-uniform changes in the locations of the discrete points must be associated with change in object shape. However, this is possible only if all objects are pinned in space. Perhaps for this reason, the concept of peri-saccadic spatial compression, which was suggested by findings from peri-saccadic visual localization studies of points (Ross et al., 1997), has been often interpreted as meaning deformed geometry. Our study did not find that the pattern of compressed localization was able to predict apparent object size. This is consistent with previous findings that solid objects located at the saccade landing point were perceived to have the same size as in the fixation condition (Matsumiya & Uchikawa, 2001; Noritake et al., 2009). Importantly, here we show that when there was mislocalization, the perceived size still did not appear to be accordingly compressed (see Figure 5) across the entire saccade range. Unlike previous studies that investigated whether objects centered at the saccade landing point (where mislocalization error is supposed to be zero) and presented within a time window (20–40 ms long) from saccade onset were perceived smaller by any extent (Matsumiya & Uchikawa, 2001; Noritake et al., 2009; Sogo & Osaka, 2005), our study specifically addresses whether the time- and location-dependent saccadic mislocalization pattern is consistent with change in perceived size. 

The dissociation we found between size perception and localization lends a direct psychophysical support to the model of two separate “what” and “where” vision systems (Mishkin et al., 1983; Ungerleider & Mishkin, 1982). According to this model, object characteristics, such as size, shape, and color, are processed in the “what” system along a ventral stream from primary visual cortex to inferior temporal cortex. From an information stance, retinal input should be sufficient for size perception. Although there are other factors that can affect apparent size in the real world, such as apparent distance (e.g., the size constancy phenomenon where object sizes appear similar at different distances although angular sizes decrease with distance) and reference (e.g., the Ebbinghaus illusion where objects look larger when next to a smaller reference), they are usually excluded from peri-saccadic perception studies. On the other hand, an object's spatial location is processed in the “where” system along a dorsal stream from primary visual cortex to posterior parietal cortex. The dorsal stream receives convergent input from other modalities. In cases of peri-saccadic visual localization, object location is computed using retinotopic information and at least an efference copy of the ocular-motor system. As the efference copy may take effect even before the eyes move (Duhamel, Colby, & Goldberg, 1992), the mis-synchronization between retinotopic image and the efference copy is believed to contribute to the peri-saccadic mislocalization (Pola, 2004), although it may not be able to fully explain the phenomenon. According to our data, the mis-synchronization factor does not appear to affect size perception. It indicates that size is not determined by visual localization. 

An interesting question raised here is how object location is coded. Is the overall location coded, for instance defined as the center point, or are some anchor points coded, for instance the corners of a rectangle? We think the former might be true. From the stance of coding efficiency, coding of one location is the minimum requirement. From the point of view of “what” and “where” systems, the occupancy of an object in space can be reconstructed by placing it at a certain location, without needing to consider how its boundary is registered to the background or reference. We assume that this might be what occurred in our subjects when they reported the perceived bars. Supportive evidence was also reported in one previous saccadic mislocalization study (Ross et al., 1997), in which some subjects reported that an object—in this case, the Sydney Opera House—in a natural picture could appear to detach from its background before saccades. More common cases in daily life are the mislocalization of flickering stimuli during eye movements. For example, the flickering LED digits on a clock in dim light during both voluntary (saccades) and passive (induced by vibration) eye movements may appear to move outside of the clock (Peli & Garcia-Perez, 2003). 

Future research could complement the current results by demonstrating a double dissociation between the “what” and “where” systems, by finding an experimental condition that affects perceived object size but not localization. A possible paradigm for such a demonstration might be saccadic adaptation, in which repeatedly displacing the target of a saccade during its execution leads to changes in saccade programming (McLaughlin, 1967). A recent investigation (Garaas & Pomplun, in press) has revealed post-adaptive changes in perceived object size, while it has been shown that such an adaptation may not affect perceived visual direction (McLaughlin, Kelly, Anderson, & Wenz, 1968). 

Acknowledgments

This work was supported in part by NIH Grants AG034553 (GL), EY12890 and EY05957 (EP), and EY019545 (MP). 

Commercial relationships: none. 

Corresponding author: Gang Luo. 

Email: [e-mail]. 

Address: 20 Staniford St., Boston, MA 02114, USA.