The reentry hypothesis: linking eye movements to visual perception

Abstract

Cortical organization of vision appears to be divided into perception and action. Models of vision have generally assumed that eye movements serve to select a scene for perception, so action and perception are sequential processes. We suggest a less distinct separation. According to our model, occulomotor areas responsible for planning an eye movement, such as the frontal eye field, influence perception prior to the eye movement. The activity reflecting the planning of an eye movement reenters the ventral pathway and sensitizes all cells within the movement field so the planned action determines perception. We demonstrate the performance of the computational model in a visual search task that demands an eye movement toward a target.

Introduction

Visual perception is proposed to rely on a pathway for object vision and one for action control (Milner & Goodale, 1993). Perception seems to be related to the activity of inferior temporal (IT) cells (Leopold & Logothetis, 1999). In visual search, it has been shown that the activity of IT cells predicts the action (i.e., the location of an eye movement [Chelazzi, Miller, Duncan, & Desimone, 1993; Chelazzi, Duncan, Miller, & Desimone, 1998]). Although the initial activation of IT neurons is largely stimulus driven with cells encoding target as well as non-target becoming activated, different populations compete for representation, and typically the cells encoding the non-target are suppressed. Such competition is assumed to be biased by a top-down signal from working memory (Chelazzi et al., 1998). A computational approach by Usher and Niebur (1996) shows that a parallel competition based on lateral interactions is sufficient to qualitatively replicate some of those findings. This interpretation suggests a view in which visual perception precedes the eye movement: Perception is used to compute the parameters of the action. For example, Desimone and Duncan (1995) speculate “at some point in time, mechanisms for spatial selection may also be engaged to facilitate localization of the target for the eye movements.” 

Other recent experiments reported that presaccadic activity in V4 increased at the location of the intended eye movement and decreased elsewhere (Moore, 1999; Tolias, Moore, Smirnakis, Tehovnik, Siapas, & Schiller, 2001). These findings suggest a view in which selective activity for perception is determined by the preparation of an action. 

This raises the question of whether these different interpretations are due to the different experimental paradigms used or if they are related to each other? In this contribution, we argue that these different findings are related to each other. 

We suggest a model that is based on reentrant processing. Visual perception and action selection operate in parallel. Areas typically involved in perception provide relevant parameters for action selection, such as an activity landscape that provides information about the location of relevant objects. Areas typically involved in action selection, however, enforce a decision that is continuously fed back to areas involved in perception. 

We now present a computational model of V4, IT, frontal eye field (FEF), and prefrontal (PF) areas to illustrate the following two aspects of visual search: (i) How does visual perception affect the eye movement? (ii) How does a saccadic decision determine perception? We use this model to simulate the visual search experiment of Chelazzi et al. (1998) and discuss the implications of the simulation results on the interpretation of the experimental data. 

Model

We model aspects of the areas V4, IT, FEF, and PF (Figure 1). Please refer to for computational details. IT, V4, and PF are subdivided into different (artificial) dimensions (e.g., “color” and “shape”). The model consists of ascending populations called stimulus (s) cells that can be primed by feedback connections, and descending populations, target (t) cells, that project the dominant patterns back into the source areas. 

Figure 1

 

A. Sketch of the simulated areas. Each box represents a population of cells. The activation of those populations is a temporal dynamical process. Bottom-up (driving) connections are indicated by a yellow arrow and top-down (modulating) connections are shown as a red arrow. B. Outline of the minimal set of interacting brain areas. The model areas are restricted to elementary but typical processes and do not replicate all aspects of these areas.

The model prefrontal cortex serves two major functions: memorizing a pattern in PFwm (working memory) cells and indicating a match of the incoming pattern with the memorized pattern in PFm cells. Thus, IT cells can drive only PFm cells when their pattern matches the prior knowledge from PFwm cells. 

We propose feature-specific top-down connections similar to those suggested within the feature-similarity (Treue & Martínez Trujillo, 1999; Saenz, Buracas, & Boynton, 2002) and biased competition (Chelazzi et al., 1993) frameworks. The idea is that feature-specific feedback emphasizes relevant features throughout the ventral pathway, so a specific interpretation of the scene becomes dominant at all levels of perception (Hamker, 1999). 

The first question is then, how does visual perception affect the eye movement? The frontal eye field is in an ideal position to control eye movements because it has connections to occipital, temporal, and parietal areas, the thalamus, superior colliculus, and prefrontal cortex (Schall, Morel, King, & Bullier, 1995a). The projections from V2 and V3 are weak, from V4 intermediate, and heavy from TEO. The neurons in the FEF can be categorized based on their responses to visual stimuli or saccade execution into visual, visuomovement, fixation, and movement cells (Schall, Hanes, Thompson, & King, 1995b). 

In the model, PFwm cells modulate visual processing via feedback into ITs according to the current goal of the task. The resulting local increase of firing in ITs cells is directed further downward by feedback from ITt cells to V4 cells. Increased local activity in V4 enhances the visually responsive neurons in the frontal eye field, so these cells reflect the task-relevance of a location. We consider visuomovement (v), fixation (f), and movement (m) cells in the model (Figure 1). The FEFv neurons receive convergent afferents from features in V4 at the same retinotopic location. FEFf cells generally inhibit FEFm cells unless threshold detection of the PFm cells indicates that the target is in the search array. In this case, the input into the FEFf cell is removed so that FEFm cells are disinhibited, and thus the mapping from sensory to motor is facilitated. FEFv cells activate FEFm cells by feedforward excitation and surround inhibition. Because there is evidence that saccades are elicited when movement-related activity in the FEF reaches a particular level (Hanes & Schall, 1996), a fixed threshold in model FEFm cells initiates a saccade. 

The model as outlined above can explain how knowledge about features of an object of interest might influence eye movement selection so an eye movement is goal directed. We now address the question of how a saccadic decision determines perception. According to our reentry hypothesis, activity in the movement planning areas is sent back to extrastiate visual areas and sensitizes cells within their movement field, so the object of interest gets dominant even before it is foveated. Recent evidence points toward the FEF as a possible origin of a reentry signal (Moore & Armstrong, 2003). In the model, the FEFm cells send a spatially organized signal to V4 and IT stimulus cells, which enhances the sensitivity of particular V4 and IT cells (Figure 1). What could be the effect of the enhancement of the sensitivity in V4 and IT? We have shown earlier that a spatially organized feedback signal into model V4 can account for known attention effects (Hamker, 2003): If the receptive field contains just one stimulus, then spatial feedback results in a multiplicative gain increase. This has been observed in the middle temporal (MT), medial superior temporal (MST), and V4 (Treue & Maunsell, 1999; McAdams & Maunsell, 1999). If two stimuli are presented within the same receptive field, then the model V4 reproduces the data of Reynolds, Chelazzi, and Desimone (1999): A bias toward one stimulus reduces the influence of the other stimulus within the receptive field. These earlier simulations have shown that a spatial bias influences competitive interactions, so processing gets focused on a specific area. We now argue that such a spatial bias can be the result of reentrant processing from the FEF. 

Results

To demonstrate the possible role of reentry, we simulated the previously mentioned memory-guided search task (Chelazzi et al., 1998) (Figure 2A). If the sample reappears in the search array, the condition is called “target present.” In this case, the monkey has to indicate the detection of the target object by selecting its location for an eye movement. In the “target absent” condition, the cue stimulus is different from the stimuli in the choice array. Now a saccade has to be withheld. 

Figure 2

 

A. Simulation of the experiment of Chelazzi et al. (1998). The objects are represented by a noisy population input, here illustrated by a snapshot. RFs without an object just have noise as input. Each object is encoded within a separate RF (illustrated by the dashed circle) of V4 cells in two simulated dimensions (only one is shown). All V4 cells are within the RF of the IT cell population. The model has to indicate a successful search, by selecting the previously shown object as the target of an eye movement. An artificial input is used to represent the stimuli such that each stimulus is equivalent to a specific population on the feature axis. B. Activity within the model areas aligned to the onset of the search array in the different conditions. The time of the eye-movement (240 ms) is indicated by the red bar on the time axis.

In the simulation, a pattern of inputs corresponding to the target was presented to the model. PFwm cells recorded this pattern to allow a goal-directed search. Prior to the onset of the search array, the feedback from active PFwm cells had already increased the baseline activity of the IT cells selective for the target (Figure 2B). When the search array appears, inputs are processed bottom-up, limited only by receptive field convergence and competitive interactions. Each cell initially encodes the presence of its preferred stimulus, but the target cell shows an early advantage due to top-down modulation from PFwm cells (Figure 2B). Between 150 and 300 ms, the cells encoding the non-target get suppressed, although the input is still present, whereas the cells encoding the target remain active. A crucial condition is the target absent condition. Both non-targets decrease their activity, but less than in the distractor suppression case. A simple winner-take-all competition would not replicate the experimental data because due to noise in the system, a non-target would be selected in the target absent condition. The simulation results even match the temporal course of activity of IT cells (Chelazzi et al., 1998) and V4 cells (Chelazzi, Miller, Duncan, & Desimone, 2001). This consistency allows us to give reliable predictions of the processing in other areas. 

Predictions

Involvement of the FEF

V4 cells are selective for feature and location. Due to feedback from IT to V4, the model predicts a feature-specific target effect in V4. Additionally, the model also predicts a spatially selective target effect in V4 due to reentry from the FEF. V4 cells project to FEFv neurons, which encode only location. Thus, an enhanced firing rate of a V4 cell is picked up by the FEF and leads to a target discrimination in the FEFv cells. Such a discrimination can be used for overt and covert search. In overt search, an eye movement is executed by the model when the activity of the FEF movement cell reaches a threshold. Covert search is possible if activity from FEF cells reenters the extrastriate visual cortex and enhances the input gain in V4 and IT in a spatially organized manner. We achieved the best fit with the data from Chelazzi et al. (1998) by implementing a reentry signal from the movement cells, because the ratio of target discrimination and distractor activity is higher than in the FEFv cells. Thus, we suggest that movement cell activity could provide a cortical reentry signal to bias competition in a spatially organized manner (Hamker, 2001). 

How can we differentiate between these two biases in V4? Prior to about 100 ms before a saccade, target effects in V4 and IT are related to feature-based target selection. After 100 ms, however, the spatial reentry signal dominates. This could explain the weak early and strong late target effects in V4 (Chelazzi et al., 2001). As can be seen from Figure 2B, the activity of the movement cells starts to rise 150 ms prior to the saccade and reaches a value of 0.1 around 100 ms prior to the saccade. At that time they start to discriminate between target and distractor. Although a comparison across different experiments can be problematic, this prediction is consistent with the observation that sensitivity enhancement in V4 occurs no earlier than 91 ms prior to saccade onset (Tolias et al., 2001). The design of the experiment of Tolias et al. (2001) suggests that the observed effect is spatial in nature. 

Error Trial

The target effect in the ventral pathway during visual search can be separated into early and late target effects (Chelazzi et al., 1998; Chelazzi et al., 2001). The model predicts that on average the late target effect occurs earlier in fast response trials than in slow response trials. Although this prediction can be tested easily by correlating the late target effect in IT and V4 with the time of eye movement, it does not necessarily prove that the late target effect is affected by the planning of an eye movement. 

A way to test the reentry hypothesis is to exploit error trials (Figure 3A), which can occur for several reasons. Error trials with a wrong target template (e.g., due to its use as a target in previous trials) would activate IT cells selective for a distractor and thus guide the eye toward an incorrect location. Such error trials are not of special interest, because they reflect just another (ill-defined) target search. Error trials with a correctly used target template would show an initial early target effect in IT with the same sign as in correct trials (Figure 3B). However, if the correct target selection in IT cells cannot be effectively transferred into the FEF (e.g., the initial selection is too weak), errors in the eye movement can occur. A model of biased competition that receives solely a feature-specific bias predicts that IT activity indicates the correct target selection throughout the trial until the eyes have shifted to the distractor. The planning of an eye movement has no effect on the competitive effects in V4 and IT. Only foveation then allows the distractor to be more actively represented. 

Figure 3

 

A. Error trial in the simulation of the experiment of Chelazzi et al. (1998). The red arrow in the search array indicates a wrong eye movement. In this error trial, noise in the FEF leads the system to a wrong location. The “apple” and “banana” are represented by a different population on the feature axis. B. Activity within the model areas aligned to the onset of the search array in the different conditions. The initial advantage of the cells encoding the target (red) in IT and V4 indicates that the target template in prefrontal areas is set correctly. The model predicts that if the FEF is planning an eye movement to a wrong location; the reentry of the FEFm activity into V4 and IT sets a bias such that the target effect in V4 and IT is lost prior to an eye movement. The time of the eye-movement is indicated by the blue bar on the time axis, because the eye-movement is directed to a distractor.

A biased competition model with a second bias from the FEF movement cells predicts that the bias from the movement cells overwrites the feature-specific bias, so the initial target effect in IT and V4 breaks down prior to the execution of the wrong eye movement (Figure 3B). The model predicts the crossing of the average activity more than 50 ms prior to the eye movement. 

We admit that the discrimination between the two above mentioned error trials is impossible at the behavioral level, and even a discrimination on the single cell level is difficult given normal neuronal fluctuations. An analysis of error trials would only be possible if we can infer within a single trial whether the monkey uses the correct target template. Thus, recordings have to be made from a sufficient high number of cells in order to average across cells with similar feature tuning (selectivity for the target or distractor). If we find a statistically significant early target effect in the population response, we have evidence that the correct target template is used. 

The search for a high number of IT cells selective for either target or distractor is time consuming and requires multiple recording techniques. Alternatively, for the purpose of an experimental evaluation with relatively simple techniques, we suggest a modified experiment. The search is identical to the discussed experiment with one target and two distractors, except in 50% of the trials the monkey is faced with an arrow that appears 50 ms after the search array. The arrow can either confirm the search by pointing to the target or reject the search plan by pointing to one of the distractors. If the arrow points to one of the distractors, the monkey is required to make an eye movement to this object. Because the arrow occurs only in a subset of trials, the monkey always has to begin with the feature search mode. When the location cue appears, the monkey starts to plan an eye movement toward the indicated location, regardless of the distractor’s identity. The initial target is no longer relevant, and one would expect the loss of feature-based bias from prefrontal areas. It is very unlikely that the monkey now starts to construct a search template using the features of the new target object, because this would require identification before selection. In these trials, a model of biased competition, which receives solely a feature-specific bias, predicts that IT activity initially shows an early slight target-distractor discrimination. Because the memorized target becomes irrelevant, one would predict that either the initial target selection remains visible in IT or that the early target effect dissolves, and as in the target absent trails in the experiment of Chelazzi et al. (1998), cells encoding target and distractor fire with the same frequency. A biased competition model with a second bias from the FEF movement cells predicts that the IT activity of a cell encoding a distractor exceeds the one encoding the previous target prior to the shift of the fovea. Similar to an analysis of error trials, this experiment helps shed light on the question of whether IT competition, and thus perception, is biased by planning an eye movement – a prediction of the reentry hypothesis. 

Discussion

It has been previously suggested that spatial attention is a consequence of a facilitation of neurons in spatial maps originating from a preparation of a movement (Rizzolatti, Riggio, & Sheliga, 1994). However, this hypothesis was not considered for explaining the target discrimination in IT (Chelazzi et al., 1998; Chelazzi et al., 2001). Chelazzi et al. (1998) hypothesize that the late divergence in activation may depend on competitive interactions within IT cortex, which are biased by top-down projections from the prefrontal cortex. Our simulation results offer support for an extended explanation of the late divergence in activation in which the movement cells in the FEF or the occulomotor system (FEF, LIP, and SC) in general facilitate the processing of the target object in the ventral pathway due to a reentrant signal. We compared our model data with the experimental data of Chelazzi et al. (1998) as well as Chelazzi et al. (2001) and found a good match of the temporal course of activity in IT and V4. This match is especially interesting because we ensured that the model agrees with our current knowledge of anatomy (Stanton, Bruce, & Goldberg, 1995; Schall et al., 1995a) and that the model FEF is consistent with experimental findings (Schall, 2002). 

We suggest that the findings of Chelazzi et al. (1993), in context with the interpretation suggested by the model of Usher and Niebur (1996), should be revisited in the light of recent findings showing presaccadic activity in V4 (Moore, 1999; Tolias et al., 2001). This is consistent with the idea that LIP, SC, and FEF participate in a distributed network to provide a spatial bias (Bisley & Goldberg, 2003). There is particular evidence for the contribution of the FEF. It was recently shown that visual responses in area V4 could be enhanced after brief stimulation of retinotopically corresponding sites within the FEF (Moore & Armstrong, 2003). If the planning of an eye movement indeed has an effect on the activity in the ventral pathway, error trials with an intact target template should reflect the suppression of the target in IT cells prior to an eye movement, as indicated by the simulations. 

We compared our model with data in which the monkey responded by making an eye movement toward the target. Chelazzi et al. (1998) report similar findings in a task where the monkey responded by pressing a lever. Our model would produce qualitatively similar results if we assume that in this task the monkey is planning an eye movement, but movement cells do not reach threshold activity. 

Assuming the FEF is indeed a direct source, we do not know which cells in the FEF provide a reentry signal. Visual as well as movement cells exhibit target selection. There are two possible models: a visual selection model and a movement plan model. In the visual selection model, the target selection in the visual cells is suggested to be responsible for the attention effects in extrastriate visual areas (Thompson, Bichot, & Schall, 1997; Murthy, Thompson, & Schall, 2001; Sato & Schall, 2003). In the movement plan model suggested here, activity of the movement cells is required to produce a reentry signal. At present it is not possible to rule out either model. A potential problem could arise in the movement plan model explaining covert attention. During fixation, movement neurons might be inhibited by fixation cells, and thus are presumably inactive, whereas visual neurons are not inhibited and therefore can provide both the attention signal that modulates visual processes in extrastriate cortex and the target selection signal to the movement neurons. However, no experiment has clearly ruled out that the movement cells are inactive during covert attention. It is possible that fixation cell activity is reduced, which in turn allows movement cells to be active but below the level that elicits an eye movement. A potential problem of the visual selection model is its low signal-to-noise ratio. Although the visual cells show a target selection, distractor activity is initially almost equally strong. If these activities would be directly fed back, spatial attention would be distributed to several stimuli. In addition, the target selection in visual cells appears very early as compared to the late occurrence of spatial attention in some psychophysical experiments (Weichselgartner & Sperling, 1987). It might be possible that the visual cells simply reflect the bias in other visual areas (e.g., V4) but do not provide the reentry signal. More experiments to investigate the predictions of these two models are necessary. 

Our simulations point toward a model in which selective activity for perception is determined by the preparation of an eye movement. Reentry in general has the potential of integrating multiple cortical areas (see Tononi, Sporns, & Edelman, 1992). We have simulated possible effects of feedback on the ventral pathway. Because the FEF also projects to MT and MST in the dorsal pathway, it might be interesting to investigate whether such reentrant processing might be responsible for perisaccadic mislocalization (Krekelberg, Kubischik, Hoffmann, & Bremmer, 2003). Given such a distributed system, what would be the major implications for explaining attention? 

There has been a long debate whether a system for attention overlaps with the one for programming saccades. For example, spatial attention is often suggested to precede a saccade, and perhaps select the endpoint. The model suggests that there is no separate spatial attention system. Activity associated to planning an eye movement reenters visual areas, such as V4 and IT. As a result, suppressive and facilitatory effects occur, commonly referred to as “attention.” Spatial attention could be interpreted as a shortcut of the actual planned eye movement. Thus, under natural viewing conditions, spatial attention and eye movement selection are automatically coordinated so prior to the eye movement, the amount of reentry is maximized at the endpoint and minimized elsewhere. This would facilitate planning processes to evaluate the consequences of the planned action. 

Conclusions

Our model predicts that reentrant processing from the movement cells in the FEF or the occulomotor system in general biases perception toward the planned action. 

Acknowledgments

The main part of this research was done in the lab of Christof Koch (Caltech). It was supported by Deutsche Forschungsgemeinschaft HA2630/2-1 and in part by the National Science Foundation (ERC-9402726). Commercial relationships: none. 

Appendix A

We now give a formal description of the model. Each connection in the model has an independent additive noise term that leads to variations in the transmission from one cell to another. 

Stimuli

Input stimuli I_d,i,x are encoded as a population of cells i determined by a Gaussian distribution at each dimension d and each location x. 

V4

At each of 6 possible locations x ∈ {1…6} and each feature dimension d, we simulate a neural population y^V4 (1), that receives its stimulus input according to Figure 2A:  

 

(1)

. 

The input is a result of bottom-up input I^↑ modulated by lateral I^↔ and top-down gain control I^↓. I_d,i,x is defined by the task. In order to generate a typical V1-like response driving the V4 cells, we modulate I_d,i,x with a synaptic depression term S_d,i,x (Hamker, in press):  

 

(2)

. 

Lateral connections act on the population input (3). The feedback type input (Figure 1) originates in ITt and FEFm. Reynolds, Chelazzi, and Desimone (2000) found that the effect of spatial attention can be best described as a contrast gain model. Attention increases the effective strength of a stimulus but less with high-contrast stimuli. Chelazzi et al. (1998) used high-contrast stimuli. We do not aim to explain possible underlying mechanisms of this effect here, but rather account for the finding by decreasing the efficiency of the reentry signal when the cell activity is higher according to  

 

(3)

with σ(a) = max(a, 0).  

 

(4)

 

Each population receives baseline inhibition B as well as short- and long-range inhibition (5). We assume that long-range inhibition is mediated by a pool of inhibitory neurons z^V4_d(t) that collect the activity of each population (6).  

 

(5)

 

 

(6)

 

IT

In our model, we do not increase the complexity of features from V4 to IT. Thus, our model IT populations represent the same feature space as our model V4 populations. The receptive field size, however, increases in our model, so all populations in V4 converge onto one population in IT. Due to this convergence, we apply a pooling function f = max_x (Hamker, 2003),  

 

(7)

. 

The overall input depends on the V4 cells that drive the population and on the feedback signals that enhance the sensitivity of IT cells (Figure 1),  

 

(8)

 

 

(9)

 

 

(10)

. 

The inhibitory components are similar to V4 except that we implemented only one IT population,  

 

(11)

 

 

(12)

. 

ITt gets input only from ITs cells (Figure 1). These cells ensure by strong competition that only a few active cellsfeed back into V4,  

 

(13)

 

 

(14)

 

 

(15)

 

 

(16)

. 

PF

The underlying circuits, which are responsible for memory and the detection of a match, can involve many regions, including subcortical areas. For simplicity, we assume a recurrent local circuit for memory, which is driven by ITs cells,  

 

(17)

 

 

(18)

 

 

(19)

. The variable I^store defines whether a pattern that fulfills should be memorized. It is externally set according to the task instruction. If a pattern is memorized, the term ensures that no other stimulus in IT can penetrate the memory. To determine whether a pattern in the visual scene is similar to the pattern in memory, we define match cells (PFm) that compare in parallel the current pattern in ITs cells with those in PFwm (Figure 1),  

 

(20)

 

 

(21)

 

 

(22)

. 

Activity rises in PFm cells only if populations in ITs and PFwm match. 

FEF

We simulate frontal eye field visuomovement neurons, which receive convergent afferents from V4 at the same retinotopic location (Figure 1). Different dimensions d add up,  

 

(23)

 

 

(24)

 

 

(25)

 

 

(26)

. 

The firing rate of these cells could be interpreted as representing the saliency or behavioral relevance of a location, whereas the saliency of each feature is encoded in the ventral pathway. Increased activity in FEF movement cells occurs when FEF fixation cells disinhibit the population (Figure 1). Such disinhibition of the fixation cells occurs when the PFm cells signify a match with the target (because the monkeys in the experiment were trained to make an eye movement only toward the target and hold fixation in the target absent condition). In addition to a feedforward excitation, the effect of the FEFv on FEFm cells is a slight surround inhibition. A strong self-excitory component allows the movement cells to ramp-up.  

 

(27)

 

 

(28)

 

 

(29)