Dynamic interactions between visual working memory and saccade target selection

Abstract

Abstract

Abstract:

Abstract  Recent psychophysical experiments have shown that working memory for visual surface features interacts with saccadic motor planning, even in tasks where the saccade target is unambiguously specified by spatial cues. Specifically, a match between a memorized color and the color of either the designated target or a distractor stimulus influences saccade target selection, saccade amplitudes, and latencies in a systematic fashion. To elucidate these effects, we present a dynamic neural field model in combination with new experimental data. The model captures the neural processes underlying visual perception, working memory, and saccade planning relevant to the psychophysical experiment. It consists of a low-level visual sensory representation that interacts with two separate pathways: a spatial pathway implementing spatial attention and saccade generation, and a surface feature pathway implementing color working memory and feature attention. Due to bidirectional coupling between visual working memory and feature attention in the model, the working memory content can indirectly exert an effect on perceptual processing in the low-level sensory representation. This in turn biases saccadic movement planning in the spatial pathway, allowing the model to quantitatively reproduce the observed interaction effects. The continuous coupling between representations in the model also implies that modulation should be bidirectional, and model simulations provide specific predictions for complementary effects of saccade target selection on visual working memory. These predictions were empirically confirmed in a new experiment: Memory for a sample color was biased toward the color of a task-irrelevant saccade target object, demonstrating the bidirectional coupling between visual working memory and perceptual processing.

Introduction

The efficient completion of most human activities depends on directing the eyes to goal-relevant objects as each object is needed in the task (Land & Hayhoe, 2001). For example, when making tea, gaze is directed in quick succession to the spoon, then to the sugar, then to the teacup, and so on. Such rapidly changing task demands require a means to flexibly configure attentional guidance so that gaze can be oriented to the appropriate object. In most theories of attention (e.g., Bundesen, Habekost, & Kyllingsbaek, 2005; Desimone & Duncan, 1995; Duncan & Humphreys, 1989), this guidance function is proposed to involve the activation and maintenance of diagnostic target features in visual working memory (VWM). Indeed, a large body of research indicates that overt and covert visual attention are biased toward objects in the visual field that match features maintained in VWM (Hollingworth & Luck, 2009; Olivers, Meijer, & Theeuwes, 2006; Soto, Heinke, Humphreys, & Blanco, 2005). 

A central, outstanding issue in the study of goal-directed vision is the locus of interaction between VWM and perceptual selection. Most theories of VWM-mediated orienting posit an initial, bottom-up stage of visual processing that is independent of VWM. In biased competition accounts (Bundesen et al., 2005; Desimone & Duncan, 1995), the first, unselective sweep of sensory processing makes contact with VWM representations maintained in prefrontal regions, which then have their effect on selection via subsequent feedback operations. In most priority map accounts (Bisley & Goldberg, 2010; Schall, 2004; White & Munoz, 2011; see also Wolfe, 1994), visual salience is computed independently of strategic biases and is integrated with goal-level information only at later stages of processing. These assumptions have been consistent with evidence that very rapidly generated saccades are influenced solely by the physical properties of the stimuli, with goal-level influences observed only for saccades with latencies greater than approximately 175 ms (Ludwig & Gilchrist, 2002; Mulckhuyse, van Zoest, & Theeuwes, 2008; van Zoest, Donk, & Theeuwes, 2004). 

In contrast with these assumptions, Hollingworth, Matsukura, and Luck (2013b) found that even the most simple and rapid forms of saccadic orienting can be modulated by the relationship between the target stimuli and the contents of VWM (Figure 1). In a dual-task paradigm, the authors had participants maintain a color value in VWM in preparation for a memory test. During the retention interval, the participant completed a saccade task, which required a reflexive, orienting saccade to a single, abrupt-onset target. The match between the color in VWM and the color of the saccade target was manipulated. Color match influenced both the latency and the landing position of these simple saccades: Saccades to matching objects were generated more rapidly and landed closer to the center of the target than did saccades to nonmatching objects. In a related study, VWM modulation was observed for saccades with mean latencies in the range of 100 to 120 ms, indicating an interaction between VWM and initial sensory processing (Hollingworth, Matsukura, & Luck, 2013a). Moreover, memory match had a substantial effect on fixation probability in a competitive context: When a distractor was present in the display, the probability that the initial saccade was generated to the target or to the distractor was controlled by which of the two objects matched the color held in VWM. 

Figure 1

 

(A) Method in Hollingworth et al. (2013b). Stimulus configurations for different phases of a trial, with the eye icon indicating horizontal fixation position during each phase. (B) Landing position and latency results replotted from the data of Hollingworth et al. (2013b). The plot shows the histogram of signed landing position errors (negative values reflect undershoot of the target; positive values reflect overshoot).

The results of Hollingworth et al. (2013a, 2013b) indicate that VWM modulates the first sweep of sensory processing through the visual system. The visual salience of an object is not determined only by its physical properties; rather, visual salience is a joint attribute of an object's physical properties and the match between those properties and the current contents of VWM. Note that in these studies, and in other studies probing VWM-guided selection (Mannan, Kennard, Potter, Pan, & Soto, 2010; Olivers et al., 2006; Soto et al., 2005), the features in VWM are orthogonal to the orienting task, allowing researchers to dissociate automatic, VWM-based biases in selection from strategic influences on orienting. Under normative conditions, however, VWM will be used to represent features of behaviorally relevant objects, and the same mechanism of early sensory modulation would support rapid, goal-directed orienting (e.g., Hamker, 2006). 

In the present paper, we present a neurodynamic model that implements the type of low-level interaction between VWM and saccade target selection suggested by the findings of Hollingworth et al. (2013a, 2013b). The model implements (1) visual perceptual processing; (2) the formation, maintenance, and retrieval of VWM representations; (3) the allocation of spatial and feature-based attention; and (4) saccade target selection and execution. Our approach uses the framework of dynamic field theory (Johnson, Spencer, & Schöner, 2008; Schneegans & Schöner, 2008), which describes the continuous evolution of activity distributions in neural populations as the basis for perception, working memory (WM), and motor behavior (Bastian, Schöner, & Riehle, 2003; Erlhagen & Schöner, 2002; Jancke et al., 1999). The architecture we present here combines previous dynamic neural field (DNF) approaches for saccade target selection and initiation with DNF models of VWM for surface features and integrates them through coupling to a low-level visual representation. 

DNF models of saccade target selection (Kopecz, 1995; Kopecz & Schöner, 1995; Trappenberg, Dorris, Munoz, & Klein, 2001; Wilimzig, Schneider, & Schöner, 2006) describe a population code representation of saccade endpoints in retinal space. When activity in the population crosses a threshold, neural interactions create a peak of activation that can stably support a saccade, with the location and timing of peak formation determining saccade metrics and latency. Activation patterns in these models are consistent with saccade-related neural activity in the superior colliculus (Anderson, Keller, Gandhi, & Das, 1998; Marino, Trappenberg, Dorris, & Munoz, 2012; Trappenberg et al., 2001), and the models have successfully accounted for quantitative details of saccade target selection and latency effects. 

DNF models of VWM use analogous population code representations to encode and maintain object surface features (Johnson, Spencer, Luck, & Schöner, 2009a; Johnson, Spencer, & Schöner, 2009b). WM content is modeled through localized activation peaks within feature space that are sustained through lateral interactions (self-excitation and surround inhibition). A key feature of these models is a close integration between feature representations in perceptual and VWM fields. These fields are distinct but densely interconnected, consistent with evidence of sensory recruitment in VWM maintenance (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009). VWM content is determined by projections from a perceptual field and remains susceptible to perceptual input after it is established. In turn, VWM exerts an influence on perceptual activity patterns. Such interactions provide a means to implement top-down influences of VWM on perceptual processing. 

In the architecture presented here, the saccade system and the VWM system are integrated by coupling them to a shared low-level visual representation, in a fashion analogous to several models of visual attention and visual search (Fix, Rougier, & Alexandre, 2011; Hamker, 2003, 2005a, 2005b). This visual representation encodes both spatial and surface features of visual stimuli, consistent with neural representations in the early visual cortex, and it provides the input to the saccade and VWM systems. In turn, the activation distribution in this low-level visual representation is modulated by feedback projections from these two systems. Feedback from the saccadic system to the low-level visual field implements spatial attention, causing enhanced sensory processing at the location of an impending saccade (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Kowler, Anderson, Dosher, & Blaser, 1995; Moore & Armstrong, 2003). Feedback from the VWM system influences feature-based attention and thereby acts on the low-level visual representations, biasing the response of the latter to facilitate the perceptual processing of memory-matching features. Feature-specific feedback is spatially broad, modulating sensory processing across the visual field, consistent with the neurophysiological literature on feature-based attention (e.g., Treue & Martinez Trujillo, 1999). 

In this approach, VWM and saccade planning are not connected directly but rather influence each other through their continuous, bidirectional interactions with the shared, low-level visual field. The same mechanism has previously been employed to model visual search behavior, where an explicit search target, retained in VWM, guides spatial attention and eye movements to task-relevant objects in a visual scene (Hamker, 2003, 2005a, 2005b). Here, we demonstrate that such interactions between VWM and perceptual processing can also explain experimentally observed metric effects in the latency and amplitudes of individual eye movements in simple saccade tasks. Moreover, the basic architectural assumption that the VWM and saccade systems are bidirectionally coupled to a common low-level visual field predicts that interaction effects should also occur in the opposite direction; that is, selection of stimuli in the model's visual field will influence existing representations in VWM. Specifically, if an object with a particular color is selected as the saccade target object, that color will automatically interact with a color already in VWM. For relatively similar colors, the local excitatory regions of the activation profiles will overlap, causing the centers of the peaks to be drawn toward each other: The color in memory will be biased toward the color of the saccade target object. We experimentally tested this model prediction. Results confirm the predicted effect. 

In the next sections, we describe the computational model, followed by two experiments. In Experiment 1, we applied the model to a psychophysical data set similar to those reported in Hollingworth et al. (2013a, 2013b) but optimized to provide a direct test of the neural dynamics in the model. The saccade mechanism, with its functionally continuous spatial representations, generates exact saccade amplitude and latency measures. This enables the model to produce quantitative fits for the experimentally observed variations in saccade metrics over trials, which are used as signatures of the interactions between VWM and saccade behavior in the psychophysical study. In addition, we used the model to capture the empirical data on memory performance in this experiment, which varied depending on the color match in the saccade task. Based on the model mechanism that explains this effect, we predicted that specific memory biases should occur in this task. This prediction was tested in Experiment 2. 

Neurodynamic model

Dynamic neural fields

The DNF model is implemented as an integrated dynamical system governed by a set of differential equations. Each DNF defines a distribution of activation u over a metric space (here color space or physical space) to reflect a neural population code representation. The change of activation at each position Display Formula and time t is described by a field equation of the form  

 

 

Here, Display Formula is a time constant, h is the field resting level, Display Formula is the external input to the field, and Display Formula is random noise scaled by noise level q. Homogenous lateral interactions in the field are described as convolution of the field output Display Formula (where Display Formula is a sigmoid function) with an interaction kernel Display Formula . The interaction kernel describes connection weights as a function of distance in feature space. It is defined as either a difference of Gaussians (with local self-excitation and surround inhibition) or an excitatory Gaussian function with a constant (global) inhibitory component. 

These interaction patterns promote the formation of localized peaks of activation, which form attractor states in the field dynamics. In sensory representations, these peaks provide stabilized detection of stimuli. In motor representations or attention control, competitive interactions between active regions produce selection decisions in which only a single activation peak can form even in the presence of multiple inputs. Finally, with sufficiently strong lateral interactions, activation peaks can become self-sustaining without external input and thereby serve as WM representations. 

Model architecture

The full model is shown in Figure 2. It consists of five DNFs, coupled to each other through mutual projections, and three discrete dynamical nodes. Two feature dimensions are represented by these fields: one dimension of horizontal stimulus position in a retinocentric reference frame with logarithmic scaling (yielding higher resolution in the foveal region), and a surface feature dimension comprising a region for color hue values and a smaller region for gray values. Note that only one spatial dimension is modeled here for simplicity, because this is sufficient to capture the relevant spatial aspects from the experiment, but that the extension to two-dimensional visual space in a DNF model is straightforward (Lipinski, Schneegans, Sandamirskaya, Spencer, & Schöner, 2012; Spencer, Barich, Goldberg, & Perone, 2012). The full field equations and parameter values are given in the Appendix. 

Figure 2

 

Model architecture. Activation profiles in response to an exemplary stimulus pattern (top) are shown for the five DNFs of the architecture (blue plots for the one-dimensional fields; color-coded activation profile for the two-dimensional field, with dark blue indicating lowest activation and dark red indicating highest activation): visual sensory field (v), feature attention field (fa), feature WM field (fm), spatial attention field (sa), and saccade motor field (sm). Arrows indicate projections between fields. Axes for the one-dimensional fields are oriented such that the feature dimensions are aligned with the corresponding feature dimension in the two-dimensional field. The activation pattern in the two-dimensional field shows the localized visual inputs (red activation peaks), effects of lateral inhibitory interactions along the surface feature dimension (dark blue vertical troughs), and excitatory feedback from the feature attention field (light blue horizontal ridges). Excitatory feedback from the spatial attention field (vertical ridges) is largely masked by the lateral inhibition. Note that the peak positions are not aligned with the stimulus positions in the display due to logarithmic scaling of the spatial dimension in the model.

At the center of the model is the visual sensory field, defined over a two-dimensional space spanned by the spatial and surface feature dimensions. This field corresponds to early visual representations in the cortex. It receives localized external input reflecting the location, color, and size of visible stimuli. These inputs induce activation peaks in the field that are stabilized against random fluctuations by lateral interactions of moderate strength. The visual sensory field provides input to two separate processing streams: the surface feature pathway and the spatial pathway. 

The surface feature pathway consists of the feature attention field and the feature WM field, both one-dimensional fields spanning the surface feature dimension. The separation of feature attention and WM is consistent with evidence that VWM and feature-based guidance can be dissociated under some conditions (Hollingworth & Hwang, 2013; Houtkamp & Roelfsema, 2006; Olivers, Peters, Houtkamp, & Roelfsema, 2011). The feature attention field receives direct input from the visual sensory field, computed by integrating over the spatial dimension. Lateral interactions with global inhibition in the feature attention field implement a moderate competition effect that favors the selection of a single activation peak. The feature attention field projects further to the feature WM field, where strong lateral interactions (local excitation and local surround inhibition) allow activation peaks to be self-sustained and persist in the absence of external input. The resting level of the feature WM field can be globally modulated to reflect the task requirements at different times during a trial. At the highest resting level, the field can form new activation peaks driven by input from the feature attention field; at the intermediate level, existing peaks are sustained but no new peaks form; and at the lowest level, all activation peaks decay. The feature WM field projects back to the feature attention field, preactivating that field for memorized color values. The feature attention field in turn projects back to the visual sensory field, where it induces ridges of activation that are localized along the feature dimension but homogenous along the spatial dimension. 

The two fields of the spatial pathway—the spatial attention field and the saccade motor field—are defined over the one-dimensional space of retinal stimulus location. Like the feature attention field, the spatial attention field is driven by input from the visual sensory field, here integrated over the surface feature dimension. Spatial attention, in turn, projects back to the visual sensory field and induces ridges of activation that are homogenous along the feature dimension. The spatial attention field also receives direct spatial input from the visual stimuli, reflecting the connectivity of the superior colliculus, which receives afferents from both the early visual cortex and more direct retinal projections via the lateral geniculate nucleus (Wurtz & Albano, 1980). Lateral interactions with global inhibition create competition between distinct active regions in this field and promote the formation of a single activation peak even in the presence of multiple inputs. The competition process can be modulated by two dynamic nodes. The fixation node drives activation in the foveal region of the spatial attention field and thereby supports sustained fixation of any currently foveated stimulus. The gaze change node, in contrast, suppresses foveal activation and thereby facilitates a shift of attention and subsequently a saccadic eye movement to any nonfoveated stimuli. 

The spatial attention field projects to the saccade motor field and induces a peak there if a sufficiently high activation level is reached (typically after competition in the spatial attention field has been resolved). The forward projection is suppressed in the foveal region such that no saccade can be induced to an already foveated location. Once activation reaches the output threshold in the saccade motor field, the evolution of activation is dominated by strong lateral interactions. These give the activation peak a stereotyped shape and time course of formation, largely independent of the properties of the input signal, and they suppress any other activation in the field such that only a single peak may exist at all times. 

From the activation peak, a saccade motor signal is generated in a space-to-rate code transformation that implements a summation with saturation model according to the classification of Groh (2001). The instantaneous saccade motor command is determined by integrating the field output over the saccade motor field, scaling the output from each point along the spatial dimension with the eccentricity that is represented by that point. This is consistent with the theory of saccade generation by Goossens and Van Opstal (2006), who proposed that every spike of a saccadic burst neuron in the superior colliculus contributes a minivector to the saccade metrics, which reflects the preferred retinal stimulus position of the neuron. The full saccade amplitude in the DNF model is then determined by integrating the instantaneous saccade motor signal over the time that a peak persists in the field. The saccade is terminated by the saccade reset node that suppresses the saccade motor peak. The activation of the node is driven by the total output of the saccade motor field (implementing a simple neural integrator), and once it reaches the output threshold, the node sends strong inhibition back to the field. During saccade execution the visual input to the system is suppressed, and when it is restored it is shifted to reflect the saccadic gaze shift. 

Experiment 1

In Experiment 1, we tested the ability of the model to account for the VWM-based modulation of saccade dynamics and metrics observed by Hollingworth et al. (2013a, 2013b). Instead of fitting existing empirical results, we reran the critical conditions from those studies in a new experiment, with several small modifications designed to optimize the relationship between the experimental paradigm and the model (see Figure 3). In particular, all stimuli appeared on the horizontal midline in the present experiment, allowing us to model the paradigm using one-dimensional attention and saccade motor fields. 

Figure 3

 

Method in Experiment 1. The structure of the task is the same as in Hollingworth et al. (2013b), with a saccade task to an abrupt-onset disk performed while a color is maintained in VWM.

At the beginning of each trial, participants saw a color square to be retained in memory for a within-category memory test at the end of the trial. During the retention interval, participants performed a saccade task. The saccade target object appeared on the horizontal midline. There were three main trial types that implemented variants of the saccade task. Target-only trials included only a single saccade target object. This object either matched the color category of the color in VWM (target match) or was drawn from a different color category (no match). The target-only trials allowed us to examine the effect of VWM match on simple orienting saccades in the absence of stimulus competition. The other two trial types included a distractor disk to probe the effect of VWM in a competitive context. The distractor was slightly smaller than the target. Either the target matched memory (target match), the distractor matched memory (distractor match), or neither matched memory (no match). On remote-distractor trials, the distractor appeared near the central fixation point in the opposite hemifield from the target. We expected saccades to be directed either to the target or to the distractor, allowing examination of VWM effects on discrete saccade target selection. On near-distractor trials, the distractor appeared in the same hemifield as the target, 2.3° closer to central fixation than the target. We expected to observe averaging saccades (i.e., the global effect), allowing us to examine the influence of VWM match on the fine-grained metrics of saccade landing position. 

Method

Participants

Twelve participants completed Experiment 1A. Eight different participants completed Experiment 1B. All were between 18 and 30 years of age with uncorrected 20/20 vision. They received course credit or were paid. 

Apparatus and stimuli

Stimuli were displayed on a 17-in. CRT monitor at a refresh rate of 120 Hz. Eye position was monitored by an EyeLink 1000 eyetracker (SR Research, Ottawa, Ontario, Canada) sampling at 1000 Hz. 

Memory task stimuli.

The memory stimulus was a 1.6° × 1.6° colored square at the center of the screen. The color category of the memory square was randomly chosen from a set of three (red, green, and blue). Within the selected category, the color value was selected randomly from a set of four similar colors. Specific color values are reported in Hollingworth et al. (2013b). 

In the memory test display at the end of the trial, two color squares were presented to the left and right of central fixation. One color was identical to the memory square presented at the beginning of the trial (target). The other was drawn from the remaining three colors in that category (foil). 

Saccade task stimuli.

In Experiment 1A, the target display contained only the target on 75% of trials (target-only trials) and a target plus a distractor on the remaining 25% of trials (remote-distractor trials). In the target-only trials (Figure 3), a single saccade target disk (0.98° diameter) was displayed to either the left or right of central fixation. Direction was selected randomly, and eccentricity was selected randomly within a range (4.6°–7.0°). The target disk's color was drawn either from the memory category (target match) or randomly from one of the two remaining categories (no match). In the former case, the color from the matching category was either an exact match for the remembered color or an inexact match that was selected randomly from the remaining three colors in the target category. On inexact-match trials, the saccade target color became the foil color in the memory display. Thus, the color of the saccade target did not predict the correct response on the memory test. 

In the remote-distractor trials of Experiment 1A, the target was accompanied by a 0.66°-diameter distractor disk presented at an eccentricity of 1.3° in the opposite hemifield from the target. In the target-match condition, the target matched the memory category (exact or inexact) and the distractor did not. In the distractor-match condition, the distractor matched the memory category (exact or inexact) and the target did not. In the no-match condition, neither the target nor the distractor matched the memory category. 

In Experiment 1B (near-distractor trials), the target display contained two objects on every trial: the 1.0° target and the 0.66° distractor. The target position was determined in the same manner as in Experiment 1A. The distractor appeared in the same hemifield as the target, centered 2.3° closer to the central fixation point than the target. The memory-match conditions were the same as in the remote-distractor trials of Experiment 1A: target match, distractor match, and no match. 

For both targets and distractors, the saccade results were identical for exact matches and inexact matches, and all analyses presented here collapse across these two types of trials except as described below. 

Procedure

Each trial began with fixation of the central cross. The color memory square was presented for 300 ms, followed by a delay of 700 ms. Then, the saccade target—and the distractor, in the distractor-present trials—was added to the display. Participants were instructed to execute a saccade to the target as rapidly as possible. They were also instructed that if a distractor appeared, it was irrelevant to the task and they should avoid fixating it. In the remote-distractor trials, the target always appeared much farther from central fixation than did the distractor and was larger than the distractor, removing any significant ambiguity in the task of orienting to the target. Similarly, in the near-distractor trials, the target was always more eccentric than the distractor and was larger than the distractor. Participants were instructed to execute a saccade to the outer object. 

When a fixation was detected in the target region, the target display remained visible for 200 ms and was then replaced with the memory test stimuli. Participants pressed one of two buttons to indicate the test square that was an exact match to the memory square. 

Participants first completed a practice session of 18 trials. In the main session of Experiment 1A, participants completed 384 trials. There were 288 trials of target only: 144 no match and 144 match. There were 96 trials of distractor present: 32 no match, 32 distractor match, and 32 target match. The match trials were evenly divided between exact and inexact match. Trials from the different conditions were randomly intermixed. In the main session of Experiment 1B, participants completed 400 trials: 100 trials of target match, 100 trials of distractor match, and 200 trials of no match. The match trials were evenly divided between exact and inexact match. 

Data analysis

A combined velocity (>30°/s) and acceleration (>8000°/s²) threshold was used to define saccades. Trials were eliminated from the analysis if the participant was not fixating within 1° of the center cross when the target stimulus appeared (7.3% of target-only trials, 6.9% of remote-distractor trials, and 4.2% of near-distractor trials). In addition, trials with saccade latency >500 ms or <60 ms were eliminated (1.8%, 2.3%, and 4.1% of the remaining trials, respectively). Finally, to eliminate saccades that were not generated as a response to the saccade task stimuli, trials in the target-only condition were eliminated if the eyes did not land within 3.0° of the center of the target (1.9% of remaining trials), and trials in the near-distractor condition were eliminated if the eyes did not land within 3.5° of the center of the target (3.8% of remaining trials). This standard could not be applied to the remote-distractor trials because saccades that failed to land near the target might have been systematically directed to the distractor. Overall, 10.8% of trials were eliminated from the target-only condition, 9.1% were eliminated from the remote-distractor condition, and 11.7% were eliminated from the near-distractor condition. Elimination of these trials altered neither the data pattern nor the statistical tests as a function of memory-match condition. 

Simulation procedure

The stimulus positions and sizes along the horizontal dimension as well as the stimulus timing from the experiments were directly adopted in the simulation. Stimulus colors in the simulation were specified as hue values, with different color categories reflected by a hue value difference of 120° and different colors within a category reflected by a hue value difference of 20°. Task instructions were reflected by external control inputs to the architecture: During the memory sample presentation, a global excitatory input was supplied to the feature WM field; during the saccade task, the gaze change node was activated and the spatial attention field preshaped to reflect knowledge of possible target and distractor locations; and during the memory test, the feature attention field was globally excited. The exact function of these inputs is described below. The memory test task, which was performed as a manual response task in the experiment, was emulated as another eye movement task. Using the same stimulus settings as in the experiment, the model performed a simple visual search for the memorized color and selected one of the two memory test stimuli by making a saccade to it. 

Model simulations were performed in blocks of 304 trials, in which the saccade target position was varied from minimum to maximum eccentricity in steps of 1 pixel (1°/30.5 in visual angle) and the direction of the color mismatch in the memory test was balanced. These blocks were repeated for the different conditions to approximate the total numbers of trials in the psychophysical study. Random noise was added to all field activations to obtain stochastic distributions of results. For Experiment 1A, the model performed a total of 4,864 trials (3,040 target-only trials, with an equal number of target-match and no-match trials; 1,824 distractor-present trials, with an equal number of trials for target-match, distractor-match, and no-match conditions). For Experiment 1B, a total of 2,432 simulated trials were performed, with an equal number of trials for the target-match, distractor-match, and no-match conditions. For all match conditions there were equal proportions of exact and inexact matches. Trials with a saccade latency of <60 ms or >500 ms were excluded from the analysis (0.03% of trials). 

Results and discussion

In this section, we first present the results of the psychophysical experiments side by side with the simulation results that the DNF model produced for the corresponding conditions. Then we explain the how the mechanisms in the model interacted to generate the observed effects. 

The primary empirical measure was the landing position of the first saccade following the onset of the saccade task stimuli. For these analyses we report the horizontal landing position relative to the saccade target center, with landing positions short of target center assigned negative values and positions beyond target center assigned positive values. A second set of analyses examined the latency of saccades. Finally, we examined accuracy on the memory task that flanked the saccade task. In general, the empirical results replicated all of the principal findings in Hollingworth et al. (2013a, 2013b). The landing position of saccades was biased toward objects that matched the color maintained in VWM. In addition, saccades were generated more rapidly when the target object matched the memory color than when it did not. 

The simulation results were analyzed in the same form as the empirical results, and both are presented together. For each of the three saccade conditions (target only, remote distractor, near distractor) the model successfully emulated the effects of memory match, both for the metrics of saccade landing position and for saccade latency. Histograms depicting the observed and simulated distributions of landing position are presented in Figure 4. 

Figure 4

 

(A–C) Observed landing position results for the three main conditions of Experiment 1: (A) target only, (B) remote distractor, and (C) near distractor. (D–F) Results from the model simulations of the three conditions. The plots show the histograms of signed horizontal landing position errors (negative values reflect undershoot or deviation toward the fixation point; positive values reflect overshoot). The blue circle and blue shaded area indicate the position and extent of the saccade target, whereas the red circle and red shaded area show the range of positions and the extent of the distractor stimulus. Bar plots in panels A, C, D, and F show mean saccade landing position relative to the target position over all valid trials, whereas bar plot insets in panels B and E show the proportion of saccades that landed within 1.5° of the target center.

Eye movement results

Experiment 1A (target-only trials):

Empirical results. The distributions of landing position relative to the target center are displayed in Figure 4A. Overall, saccades tended to undershoot the target position. Mean landing position was reliably closer to the target in the target-match condition (−0.31°) than in the no-match condition (−0.40°), t(11) = 4.38, p = 0.001. In addition to the effect on landing position, there was a reliable latency advantage for the target-match condition (140 ms) over the no-match condition (146 ms), t(11) = 3.20, p = 0.008. 

Simulation results. The simulations reproduced the differences between target-match and no-match conditions for both saccade amplitudes and latencies. The distributions of landing position are displayed in Figure 4D. Mean landing position of simulated saccades was −0.42° in the target-match condition versus −0.47° in the no-match condition (p = 0.01). Simulated saccades in the match condition were faster on average, with a mean latency of 149 ms compared with 160 ms in the no-match condition (p < 0.001). 

Experiment 1A (remote-distractor trials):

Empirical results. The distributions of landing position relative to the target center are displayed in Figure 4B. Because saccades were typically directed either to the target object or to the distractor, instead of reporting absolute landing position, we analyzed the proportion of saccades that were directed to the target (i.e., that landed within 1.5° of target center). Relative to the no-match baseline, the probability that the saccade was directed to the target increased when the target matched memory and decreased when the distractor matched memory. In the target-match condition, 93.6% of saccades were directed to the target (with 0.6% of saccades landing within 1.5° of the distractor). This was significantly greater than the 79.6% of saccades that were directed to the target in the no-match condition (with 5.1% of saccades landing within 1.5° of the distractor), t(11) = 3.62, p = 0.004. In the distractor-match condition, only 40.1% of saccades were directed to the target (with 28.8% of saccades landing within 1.5° of the distractor), which was reliably lower than the percentage in the no-match condition, t(11) = 12.7, p < 0.001. 

The latency analysis was limited to those trials on which the eyes landed within 1.5° of the center of the target. This eliminated trials on which the saccade was incorrectly oriented to the distractor. Mean latency of the initial saccade was shorter in the target-match condition (172 ms) than in the no-match condition (199 ms), t(11) = 5.85, p < 0.001. Mean latency for the distractor-match (202 ms) and no-match conditions did not differ, t(11) = 1.13, p = 0.28. 

Simulation results. Simulations of the model again reproduced this pattern of results. The distributions of landing position are displayed in Figure 4E. In the target-match condition, the first saccade landed within 1.5° of the target in 89.3% of trials. This proportion was significantly higher than in the no-match condition (69.4%, χ² = 80.3, p < 0.001). In the distractor-match condition, this proportion fell further to 39.1% of trials (significantly lower than in the no-match condition, χ² = 125.2, p < 0.001). The simulation results differ from the experimental data in that the majority of trials not landing in the vicinity of the target landed within 1.5° of the distractor (8.7% of trials for target-match, 28.8% for no-match, and 60.7% for distractor-match conditions). Mean latency of simulated saccades that landed within 1.5° of the target was 168 ms in the target-match condition. This was significantly shorter than in both the no-match condition (192 ms, p < 0.001) and the distractor-match condition (181 ms, p < 0.001). Unlike in the experiment, the latency difference between the no-match condition and the distractor-match condition also reached significance (p = 0.001). 

Experiment 1B (near-distractor trials):

Empirical results. Landing position distributions are displayed in Figure 4C. As in Experiment 1A, a memory match reduced the distractor effect when the target matched memory and increased the distractor effect when the distractor matched memory. In the no-match condition, saccades tended to land between the target and the distractor, providing a baseline measure of the averaging effect. When the target matched the remembered color, the distribution of landing position shifted toward the target. Mean landing position was closer to the target in the target-match condition (−1.07°) than in the no-match condition (−1.27°), t(7) = 4.98, p = 0.002. When the distractor matched the remembered color, the distribution of landing position shifted toward the distractor. Mean landing position was farther from the target in the distractor-match condition (−1.48°) than in the no-match condition, t(7) = 4.28, p = 0.004. 

Mean latency in the target-match condition (155 ms) was reliably shorter than in the no-match control condition (161 ms), t(7) = 2.73, p = 0.03. Mean latency for the distractor-match (160 ms) and no-match conditions did not differ, t(7) = 0.80, p = 0.45. 

Simulation results. The model simulations for this experiment likewise produced averaging saccades whose metrics were influenced by color match. The distributions of landing position are displayed in Figure 4F. Mean saccade landing position was −0.92° in the no-match conditions. In target-match trials, it was shifted reliably toward the target location (−0.75°, p = 0.001), whereas in the distractor-match condition landing position was shifted toward the distractor location (−1.10°, p < 0.001). Mean latency of simulated saccades was 149 ms for the target-match condition, which was significantly shorter than in both the no-match (161 ms, p < 0.001) and distractor-match (159 ms, p < 0.001) conditions. Differences between these latter two conditions were not significant (p = 0.07). 

Color memory results

Experiment 1A:

Empirical results. Mean accuracy on the two-alternative, forced-choice color memory test was 79.8%. For color memory performance in target-only trials, there was no effect of whether the target matched or did not match the color category in memory in a one-way analysis of variance (ANOVA) over the match manipulation (target match, no match), F(1, 11) = 1.61, p = 0.23. For distractor-present trials, there was no effect of whether the target, distractor, or neither matched the color category in memory in a one-way ANOVA over the three levels (target match, distractor match, no match), F(2, 22) = 1.13, p = 0.34. 

However, there were differences in memory performance as a function of whether the matching stimulus was an exact match for the remembered color value or an inexact match. In the target-only trials, memory performance was significantly higher when the saccade target was an exact match (84.9% and 76.8%, respectively), t(11) = 3.85, p = 0.003. On distractor-present trials, when the target matched the memory category, there was a numerical advantage for exact-match trials versus inexact-match trials (84.5% and 78.1%, respectively), t(11) = 1.62, p = 0.13. When the distractor matched the memory category, there was a reliable advantage for exact-match trials versus inexact-match trials (89.3% and 72.0%, respectively), t(11) = 3.60, p = 0.004. 

Simulation results. The emulation of the memory test in the model yielded qualitatively similar results. Mean accuracy over all trials was 85.5%. There was an overall effect of match manipulation, with performance higher in no-match trials (88.1%) than in match trials (83.5%; χ² = 20.7, p < 0.001). The decreased performance in the match condition was driven by low performance in the trials with inexact color match (77.0% compared with 89.9% for exact match; χ² = 48.9, p < 0.001). This difference between inexact and exact color match was also observed individually for target-only trials (77.0% compared with 90.3%; χ² = 49.0, p < 0.001), distractor-present trials when the target matched the memory category (74.7% compared with 88.2%; χ² = 18.3, p < 0.001), and distractor-present trials when the distractor matched the memory category (79.6% compared with 90.8%; χ² = 15.1, p < 0.001). 

Experiment 1B:

Empirical results. Mean accuracy was 75.0%. There was a reliable effect of whether the target, distractor, or neither matched the color category in memory in a one-way ANOVA over the three levels (target match, distractor match, no match), F(2, 22) = 4.99, p = 0.02. This effect was attributable to higher memory performance in the target-match condition (77.5%) than in the distractor-match condition (71.3%), t(7) = 3.49, p = 0.01. The advantage for the target-match condition could have arisen because better accuracy reduced the need for a corrective saccade, reducing the memory retention interval and the perceptual interference generated by each fixation. However, accuracy on the memory test had no influence on the key measures of saccade behavior. Saccade analyses limited to trials on which the memory response was correct yielded precisely the same pattern as reported above. 

As in Experiment 1A, there was an effect of whether the saccade task stimuli were an exact match for the remembered color value or an inexact match. When the target was an exact match, memory performance was numerically higher than when the saccade target was an inexact match (80.4% and 74.6%, respectively), although the effect did not reach statistical reliability, t(7) = 1.87, p = 0.10. There was a reliable effect of exact/inexact match for the distractor-match trials (75.5% and 67.2%, respectively), t(7) = 3.55, p = 0.009. 

Simulation results. In the simulation results for the memory test, mean accuracy was 85.8%. There was no reliable effect of match condition. In the target-match condition, performance was reliably higher if the match was exact (92.1%) than if it was inexact (77.3%, χ² = 25.7, p < 0.001). For the distractor-match condition, there was the same tendency in the performance values (88.2% compared with 84.2%, respectively), but the difference did not reach significance (χ² = 1.99, p = 0.16). 

Model mechanism for the remote distractor paradigm

We describe the model mechanism for the interactions between VWM and saccade behavior in detail for the remote distractor paradigm (Figure 5). Each trial begins with the presentation of the memory sample at the central and initially fixated location. The stimulus induces a localized activation peak in the visual sensory field, reflecting stimulus position and color (Figure 5A). This peak in turn provides input to both the surface feature pathway and the spatial pathway. 

Figure 5

 

Evolution of activation patterns during simulation of the remote distractor paradigm (distractor-match condition). Fields are shown as in Figure 2. Random noise in field activations is turned off to show typical activation time course. (A) The memory sample stimulus is presented and induces a peak for its color and position in the visual sensory field. This provides input to the feature attention field and, from there, to the feature WM field. This latter field receives a global boost input during the memory sample period and can form a peak for the presented color. This peak remains self-sustained after the stimulus is removed. An activation peak is also induced in the spatial attention field, but it does not project further to the saccade motor field because the stimulus is already foveated. (B) After a delay with only a fixation stimulus (white cross) present, the saccade target and distractor stimuli are presented. Peaks for these three stimuli in the visual sensory field project to both feature attention and spatial attention fields. The feature attention field is preshaped by feedback input from the feature WM field, such that the distractor stimulus that matches the memorized color can induce a peak here more quickly than the saccade target stimulus. (C) Feedback from the feature attention field strengthens the peak for the distractor stimulus in the visual sensory field, and this peak in turn projects more strongly to the spatial attention field. The competition in the spatial attention field is consequently decided in favor of the distractor, and a single activation peak forms at the distractor location. This peak is now strong enough to also induce a peak in the saccade motor field via the feedforward projection, and this saccade peak acts as motor signal that drives an eye movement to the distractor.

In the surface feature pathway, an activation peak for the stimulus color forms first in the feature attention field, which then projects to the feature WM field. In addition, the feature WM field receives an external control input during the memory sample period of the trial, which globally raises the field's activation level. This input reflects the task instruction to memorize the presented color. It allows the localized input from the visual stimulus to drive the field activation beyond the output threshold and induce an activation peak. In the spatial pathway, input is projected to the central part of the attention field and produces a peak there. However, this part of the field does not project further to the motor field (the projection is inactive in this region since a saccade cannot be made to an already foveated target), such that no saccade motor behavior is induced. 

For the following delay period, the memory sample stimulus is turned off. The stimulus-induced activation peaks in the visual sensory field, the spatial attention field, and the feature attention field disappear. The peak in the feature WM field remains self-sustained through the lateral interactions in the field. This peak projects back to the feature attention field. In this field, the region in hue space that matches the memorized color is preactivated, but the activation is too weak to produce significant output and generate a stabilized activation peak. Consequently, it does not produce any noticeable feedback to the visual sensory field. In place of the memory cue, the smaller fixation cue is now activated and induces weak activation peaks, first in the visual sensory field and then in the feature and spatial attention fields. 

In preparation for the upcoming saccade task, the spatial attention field is preshaped by constant input that reflects the task instructions: The spatial regions where the target stimulus can appear are preactivated, whereas the distractor regions are inhibited. The spatial attention field is additionally modulated by the gaze change node, which is activated to reflect the task instruction that an eye movement should be made as quickly as possible when the target appears. The gaze change node suppresses activation at the center of the spatial attention field (the fixation region) and thereby facilitates rapid saccade generation to extrafoveal stimuli. 

When the target and distractor stimuli are presented, they induce peaks in the visual sensory field (Figure 5B). These provide localized inputs to the spatial attention field, one on either side of the fovea, and a competition process between the active regions in this field ensues. In this competition, the preshaping of the field as well as the stimulus size favor the target stimulus, but the distractor stimulus has the advantage of being closer to the fixation point, leading to a stronger input due to the logarithmic scaling of the fields. Overall, the target stimulus has a slight advantage over the distractor, and an activation peak in the spatial attention field is more likely to form at the target location. Nonetheless, on some trials the distractor stimulus prevails due to random noise in the field activation, leading to the overall result of 69% of saccades made to the target if no color match occurs. 

A color match can bias this competition process in the spatial pathway by acting on the representation in the visual sensory field. When activation peaks for the target and distractor stimulus first form in the visual sensory field, they project localized inputs to the feature attention field reflecting the stimulus colors. If either color matches the color of the memory sample (in Figure 5B, the green distractor stimulus), the corresponding input coincides with the preactivated region of the feature attention field and can very quickly induce a strong activation peak at this location. This peak then provides earlier and stronger feedback to the visual sensory field than it would otherwise, thereby strengthening the representation of the matching stimulus in this field. At the same time, due to global inhibition in the feature attention field, the formation of a color peak for the other stimulus is hindered, and feedback for it is consequently reduced. These interactions along the surface feature pathway change the relative strengths of the peaks in the visual sensory field and thereby influence the ongoing competition process in the spatial attention field. In effect, a stimulus that matches the memorized color is more likely to prevail in the attentional competition and be selected as a saccade target (Figure 5C). This creates the saccade target selection effects in this paradigm. 

The processes of competition and selection occur simultaneously in the spatial attention field and the feature attention field, continuously influencing each other by modulating the representation in the visual sensory field. The decision for one stimulus as the saccade target can therefore be described as a coupled selection process in both of the attention fields. This is also reflected in the saccade latencies in the model. If spatial biases (from task instructions and physical stimulus properties) and feature biases (from memory color match) favor the same stimulus, the competition is resolved more quickly and saccade latencies are reduced. This is the case in the target-match condition. In contrast, in the distractor-match condition, the biases in the feature and spatial pathways are in conflict with each other, such that it takes longer to resolve the competition and select a saccade target. 

Model mechanism for target-only and near-distractor trials

The effects of a color match, both on saccade latency and saccade amplitude, are still observable if only a single target stimulus is presented. As in the remote-distractor trials, a color match modulates the representation in the visual sensory field during the saccade task and strengthens the activation peak for the target stimulus. Consequently, a stronger input is provided to the spatial attention field, which decreases the time it takes to overcome the fixation peak and to produce an activation peak in the saccade motor field, thus decreasing saccade latency. The cause for the difference in saccade amplitude is more subtle. Unlike in the other conditions, it cannot be explained by a change in the spatial distribution of activation because this is not affected by the feature match. However, it emerges in the model as an effect of incomplete normalization against stimulus intensity in the space-to-rate code transformation for saccade generation. As stated above, the strong interactions in the saccade motor field make the time course of peak formation largely independent of the characteristics of the field input. However, a small effect of input strength remains because an activation peak supported by strong input takes longer to be extinguished by the saccade reset node and consequently can produce a saccade signal for a slightly longer duration. Since the field output is integrated over the whole duration that a peak is present to compute the saccade amplitude, the color-match condition produces slightly longer saccades for equal stimulus eccentricities. 

In the global effect paradigm, averaging saccades consistent with the experimental results emerge in the model: Activation peaks from the target and distractor stimulus merge to a single active region in the spatial attention field due to broad feedforward projections and lateral interactions. The peak in the saccade motor field then forms near the center of this region. Depending on the memory color match, the center of mass for this active region can be shifted toward either the target or the distractor position, explaining the effects on saccade metrics found in the experimental data. The model also reproduces the lower saccade latencies in the target-match condition. Here, the effects of the larger target stimulus, preshaping of the spatial attention field, and modulation by feature memory all converge on the same area, leading to faster formation of the motor peak. Overall, saccade latencies are lower in the near-distractor paradigm compared with the remote-distractor paradigm because the two stimuli partly act together to produce a saccade instead of competing with each other. 

Memory performance in the model

The memory test, which was performed as a manual forced-choice task in the psychophysical study, was emulated in the model by another saccade task that takes the form of a simple feature search. The two memory test stimuli were presented equidistantly from the fixation point, and the model had to perform a saccade to the one that matched the memorized color. In preparation for this task, the gaze direction of the model was artificially set back to the fixation point to avoid any biases from the preceding saccade task. Then the activation levels of the feature attention field and the feature WM field were globally increased. Through this change, the feedback input from a peak in the feature WM field is sufficient to induce a stabilized activation peak in the feature attention field rather than just a subthreshold preshaping of its activation pattern. This brings the system into a mode of operation that implements an explicit visual search by providing direct modulatory input to the visual sensory field and thereby biasing spatial selection decisions. The basic mechanism is the same that created the interaction effects in the saccade task, but the boosting of the feature attention field makes feature match the dominant factor in saccade target selection. 

The model's performance in the memory test is limited by the fact that the colors of the target and foil stimulus are similar, such that the activation peaks induced by them in the visual sensory field are close to each other along the surface feature dimension. The modulatory input from the feature attention field therefore strengthens not only the activation for the target stimulus but also, to a lesser degree, the activation for the foil stimulus. Under these conditions, a correct saccade to the target stimulus is made in the majority of trials, but due to random noise in the fields, a selection of the foil stimulus occurs on some of the trials even if the color of the original memory sample is retained accurately in the feature WM field. Note that during the preceding saccade task, the same partial overlap in the feedback projections for similar colors is the reason that an inexact color match still has a biasing effect on saccade metrics. 

Performance in the memory test is further affected by inaccuracies in the WM representation. In the absence of external input, the self-sustained peak in the feature WM field constitutes a line attractor of the field dynamics. This means that the activation pattern is stabilized by the lateral interactions against perturbations that decrease or increase its activation level (so that the activation does not collapse or expand over time), but it is not stabilized against shifts along the feature dimension (Wu & Amari, 2005). Noise in the field can therefore lead to random drift of the peak position over time (Figure 6). Drift can also be induced by visual stimuli during the saccade task, which can project input to the feature WM field due to the continuous coupling between the fields. The effect of such inputs depends on their location in feature space. If the stimulus color is categorically different from the memorized color, the input location is distant in feature space from the peak position and does not affect the peak (Figure 6A). Random drift occurs in the same way as if no input were present. If the stimulus color matches exactly the memorized color, it stabilizes the memory peak at its initial location (Figure 6B). This reduces random drift and further stabilizes the peak against noise-induced collapse. As a result, the feedback input from the feature WM field during the memory test is more accurately centered on the correct stimulus, and memory test performance is increased. 

Figure 6

 

Effect of the saccade task on the feature WM representation in the model. The figure shows a section of the feature WM field (blue) with a self-sustained activation peak and the input that the field receives from the feature attention field when a saccade target stimulus is presented (green). (A) In the no-match condition, the input is distant from the activation peak in the space of hue values and does not interact with it. Random drift in the peak location can occur due to noise in the field (not shown). (B) In the case of an exact match, the stimulus input stabilizes the position of the activation peak and reduces random drift. (C) For an inexact match of saccade target color to memorized color, the input overlaps partly with the memory peak and induces stronger drift of the peak toward the stimulus color.

In contrast, in the case of an inexact color match in the saccade task, the input from the visual stimulus overlaps partly with the WM peak (Figure 6C). The self-sustained peak is pulled toward the region of greater activation and shifts in the direction of the input location. The effective shift of the peak is relatively small in the present task since the saccade task stimuli are visible for only a short time, but it is sufficient to affect memory performance. Since the color for the inexact match in the saccade task is always the same as the foil color in the memory test, the memory peak is pulled to a position in feature space that is intermediate between the stimulus colors in the memory test. This reduces the biasing input for the correct stimulus and increases the bias for the foil stimulus, leading to a larger proportion of incorrect choices. 

To quantify these effects, we determined the position of the activation peak in the feature WM field (as the center of mass of the field output) in each trial immediately before the memory test. We present here only the results for target-only trials from Experiment 1A to allow a direct comparison with the empirical results of Experiment 2 below, but collapsing data over all trials from Experiments 1A and 1B yields qualitatively equal results. For the analysis, trials are classified into no match, exact match, and inexact match. Trials in which no self-sustained activation peak was present at this time (due to failure to form an activation peak or collapse of the peak before the memory test) were excluded from the analysis (0.7% of trials for no match, 0.7% for exact match, and 0.4% for inexact match). Consistent with the mechanism described above, a systematic bias was found in the inexact-match trials in that mean peak position was shifted toward the stimulus color (mean deviation from the memorized hue value toward stimulus hue value was 3.4°, significantly greater than zero; p < 0.001). For exact-match and no-match trials, no systematic bias was observed (mean peak position relative to memorized hue value was 0.19° for no match and −0.10° for exact match, not significantly different from zero; p = 0.13 and p = 0.49, respectively). Moreover, the standard deviation in the position of the memory peak was lower in the exact-match trials (3.9°) than in the no-match trials (4.8°) and inexact-match trials (5.1°), consistent with a stabilization of the memory peak by an exactly matching saccade stimulus. 

This effect in the model suggests that processing of the saccade target color can bias the content of VWM, consistent with the results of the Experiment 1 memory test. The model is explicit in the mechanism by which this occurs: Interactions between color peaks serve to pull the VWM color peak toward the currently perceived and attended saccade target color. Moreover, the model predicts that this effect should be observed only for relatively similar colors because the interaction will be limited to colors whose activation gradients overlap. Finally, the model predicts differences in the variance of the color response when the saccade target color is an exact versus inexact match to the memory color. An exact match reduces random fluctuations in the color value, providing additional sensory input to stabilize the memory peak at the value corresponding to the remembered object. Thus, the model predicts a reduction in the variance of the color memory in the exact-match condition compared with the inexact-match condition. In Experiment 2, we tested these predictions using a color memory test that allowed us to estimate the absolute hue value maintained in memory on each trial. 

Experiment 2

Experiment 2 used a continuous color recall procedure (Zhang & Luck, 2008) to provide a trial-by-trial estimate of the color value retained in VWM and the potential modulation of that memory representation by the process of executing a saccade to the target. The events in a trial were similar to those in the target-only condition of Experiment 1A and are illustrated in Figure 7A. Participants saw a sample color chosen randomly from a circular hue, saturation, value (HSV) color space. The sample color was followed by a saccade target object drawn in a color that was the same as the sample color (same-color condition), that differed from the sample color by 20° in hue space (similar-color condition), or that differed from the sample color by 165° in hue space (dissimilar-color condition). Although it is not possible to establish precise correspondences between the colors in Experiment 1 and the color space used in Experiment 2, the similar-color condition was designed to generate color differences roughly equivalent to those between inexact-match colors in Experiment 1. Moreover, it exactly matches the distance in hue space used for the inexact-match condition in the simulations. After fixating the saccade target, a color wheel stimulus appeared. Using a mouse cursor, participants marked the location on the color wheel corresponding to the remembered color. We predicted that, consistent with the model's simulation of the memory effects in Experiment 1, memory for the sample color would be biased toward the saccade target color in the similar-color condition, as it was expected that the activity gradients corresponding to the two would overlap. No color bias was expected in the dissimilar-color condition. In addition, because an exact match provides additional sensory input at the remembered value, reducing fluctuations in the memory peak, the model predicts a reduction in the variance of memory responses for the same-color condition compared with the similar-color and dissimilar-color conditions. 

Figure 7

 

(A) Method in Experiment 2. The task structure is the same as in Experiment 1, but a continuous recall procedure is used instead of the two-alternative forced-choice task to assess color memory. After the saccade task, a color annulus is presented and participants have to indicate the hue value of the memorized color by moving a cursor. (B) Distribution of color responses in the same-color, similar-color, and dissimilar-color conditions. Data were normalized so that errors in the direction of the saccade target color (in color space) were coded as positive and errors in the reverse direction were coded as negative.

Method

Participants

Sixteen new University of Iowa undergraduates participated for course credit. All reported normal or corrected-to-normal vision. 

Stimuli

The stimuli were the same as in Experiment 1A, with the following exceptions. The memory square color was selected randomly from a circular HSV color space. The hue value varied from 1° to 360°, with brightness and saturation values fixed at 70%. The saccade target object subtended 1.31°. The color of the saccade target object was identical to the memory color or differed from the memory color by either 20° or 165° in color space. The memory test display consisted of a centrally presented color annulus with an inner radius of 2.46° and an outer radius of 5.74° and a plus-sign cursor. The orientation of the color annulus was selected randomly on each trial from a set of 360. 

Procedure

The stimuli were the same as in Experiment 1A until the memory test. In the test display, the color annulus was presented with a central, plus-sign cursor. The participant moved the cursor with the mouse and clicked on the color annulus location corresponding to the original memory color. This terminated the trial. 

Participants completed a practice session of six trials followed by the main experiment session of 444 trials—148 in each of the three color-match conditions (same, similar, and dissimilar). In the latter two conditions, trials were evenly divided between clockwise and counterclockwise difference. Trials from the various conditions were randomly intermixed. 

Data analysis

Trials were eliminated from the analysis if the participant was not fixating within 2° of the center cross when the target stimulus appeared (3.9%). Trials with saccade latency >500 ms or <60 ms were eliminated as outliers (4.1% of the remaining trials). Trials were eliminated if the eyes failed to land within 3.0° of the center of the saccade target object (2.1% of the remaining trials). Finally, trials with color responses greater than 85° from the memory color were eliminated as errors (1.0% of the remaining trials). 

Results and discussion

For the color memory data, each color response within the 360° space was coded as a deviation from the correct memory color (0°). For similar-color and dissimilar-color conditions, the color responses when the target color was clockwise and counterclockwise from the memory color were collapsed by coding angular deviations toward the saccade color as positive values and deviations away as negative values. In the same-color condition, the saccade target color was the same as the memory color, and thus there could have been no systematic bias toward or away from the saccade target color. For this condition, the deviation value from each trial was randomly assigned either a positive value (deviation toward) or a negative value (deviation away). This allowed us to assess the variability in same-color responses using the same measure as that used for the similar-color and dissimilar-color conditions. 

Figure 7B shows the distributions of color response deviations in the three main conditions. The key measure was mean deviation toward the saccade target color. Given that same-color trials were randomly assigned to dummy “toward” and “away” conditions, there should have been no significant deviation from zero, and indeed there was not. Mean deviation was 0.09°, t(15) = 0.59, p = 0.57. For the dissimilar-color condition, there was no significant bias toward the saccade target color. Mean deviation was −0.15°, t(15) = −0.88, p = 0.39. However, in the similar-color condition, there was a significant bias toward the saccade target color. Mean deviation was 1.9°, t(15) = 4.1, p < 0.001. Color bias in the similar-color and dissimilar-color conditions differed significantly, t(15) = 4.7, p < 0.001. Thus, executing a saccade to a target that was relatively close in color space to the remembered color interacted with memory to bias the color response 1.9°, on average, toward the saccade target color. This is comparable with the bias of 3.4° observed by reading out the memory peak in the simulations of Experiment 1 (note that these simulations differ from Experiment 2 only in that they do not explicitly model the response via the color wheel). 

The relationship between the memory and saccade target colors also influenced the precision of the memory representation. As predicted by the model, mean standard deviation was reliably smaller in the same-color condition (12.4°, compared with 3.9° in the simulations) than in the similar-color condition (14.2°, compared with 5.1° in the simulations), t(15) = 20.2, p < 0.001. Unlike in the simulations, mean standard deviation was also reliably smaller in the similar-color condition than in the dissimilar-color condition (15.9°, compared with 4.8° in the simulations), t(15) = 5.65, p = 0.03. The standard deviations here are overall higher than in the simulations, which may be explained by additional variability introduced by the response process that is not captured in the model. 

In summary, the main predictions of the model regarding the influence of saccade target selection on color memory were confirmed. When the saccade target object was a color with a value relatively similar to the color in memory, the color memory was systematically shifted toward the saccade target color. In addition, when the saccade target color was the same as the memory color, the additional perceptual input at the remembered value reduced variability in memory, as indicated by a reliably smaller standard deviation in the response distribution for the same-color condition relative to the similar-color and dissimilar-color conditions. We also observed a difference in variability between these latter two conditions, which was not predicted by the model. Future work will be needed to investigate the source of this difference. 

General discussion

We have presented a neurodynamic model of saccade target selection, VWM, and their bidirectional interactions. In recent empirical work, features maintained in VWM have been shown to modulate the initial visual salience of perceptual stimuli to bias attention and gaze toward memory-matching objects (e.g., Hollingworth et al., 2013b). In addition, we have demonstrated that the selection of an object as the saccade target biases the value of color representations in VWM toward the saccade target value (Experiment 2 of the present study). Taken together, the results show that VWM cannot be adequately described through passive and static representations; rather, VWM is active and continuously coupled to attentional processes that control saccade target selection. Our model reflects this view by structuring VWM and saccade target selection as a dynamical system comprising active and continuously interacting representations. In particular, bidirectional interactions are realized by coupling VWM and saccade target selection systems to a shared, low-level sensory field, implementing feature-based and spatial attention effects on sensory processing. The quantitative fits of behavioral data demonstrate that the interactions between spatial and surface feature representations can indeed account for the observed dynamic and metric changes in saccade behavior. Although such model fits can never prove the validity of a model, they show that the conceptual explanation is viable and consistent and does not contain any hidden assumptions or undetected conflicts. 

The neurodynamic model presented here is related to different previous lines of modeling work. It brings together separate DNF accounts of saccade planning and of VWM for spatial locations or surface features. In addition, it shares important features with previous work on the interactions of feature and spatial attention that have been used to explain the mechanisms underlying visual search (Hamker, 2003, 2005a, 2005b). In the next sections, we first discuss the relationship between the present model and earlier models of saccade planning and VWM. Then, we discuss the relationship with models of visual search. 

DNF models of saccade behavior describe the neural process underlying saccade target selection and saccade initiation as the formation of an activation peak in a field defined over retinal space (Kopecz & Schöner, 1995; Marino et al., 2012; Trappenberg et al., 2001; Wilimzig et al., 2006). This mechanism has been used to explain saccade latency effects in the gap-step-overlap paradigm (Kopecz & Schöner, 1995) and latency effects caused by the presence of distractors (Trappenberg et al., 2001). Averaging saccades in the presence of a closely spaced target and distractor have been explained through merging of activation peaks (Trappenberg et al., 2001; Wilimzig et al., 2006), a mechanism used in the present study to reproduce the behavioral data for the near-distractor paradigm. The DNF approach also provides a straightforward way to integrate task-related, top-down and stimulus-driven, bottom-up information in saccade planning by providing separate inputs with different characteristics to the same field (Kopecz & Schöner, 1995; Trappenberg et al., 2001). 

In the details of the saccade mechanism, the spatial pathway in the present model is closest to the implementation of Trappenberg et al. (2001). Both models use separate layers for saccade preparation and initiation (here the spatial attention field and saccade motor field), with the fixation activity present in the preparatory layer. This earlier model was presented explicitly as a model of saccade-related activity in the superior colliculus, whereas in our view the system also integrates functionally analogous aspects of the parietal cortex and frontal eye field activation. A significant innovation in the current work is the implementation of a space-code to rate-code transformation that generates a dynamically changing motor signal from the activation distribution in the DNF. This allows us to model the actual saccade execution (including the resulting shift in the visual image) and saccade termination as a result of neural dynamics, whereas previous models described only the processes leading up to the initiation of the saccade. The extended mechanism provides detailed saccades metrics and is in particular critical for capturing saccade amplitude effects in the target-only trials, which cannot be reproduced by simply reading out the position of the saccade motor peak. 

DNF models of VWM have been used to explain various psychophysical results concerning VWM capacity and change detection performance (Johnson et al., 2009a, 2009b). These models use self-sustained activation peaks as the memory substrate and assume a continuous coupling of WM and perception, which was also implemented in the current model. The coupling of WM to perceptual input makes the memory representations susceptible to change by subsequent perceptual states. For example, in the domain of spatial WM, Schutte and Spencer (2009) used a multilayered DNF model to explain delay-dependent drift in spatial recall estimates relative to a perceived frame of reference. Notably, young children show biases toward a perceived midline axis. This bias is modulated by the distance between the remembered location and the axis, and bias magnitude increases systematically with increasing memory delay (see also Schutte & Spencer, 2010). These modulations of spatial memory by perceived environmental structure are mechanistically similar to the metric drift of color memory in Experiment 2, when the saccade target was relatively close in color space to the memory color, suggesting that perceptual coupling and perceptually induced drift may be fundamental properties of visual and spatial WM systems. 

One significant difference in these previous DNF models of VWM compared with the present feature pathway is the presence of a contrast layer in place of the attention field (Johnson et al., 2009a, 2009b). This contrast layer receives excitatory sensory input and drives activation in the WM layer but is inhibited by feedback from the WM peaks. It thereby yields an active signal when a difference is detected between memorized and current visual input. The feature attention field in the present architecture instead is coupled in a purely excitatory fashion to the WM field. We consider these different connection patterns to reflect different functional aspects of visual processing, one enabling parallel change detection and the other responsible for the selection of individual items for focused processing. A recent scene representation model demonstrates that these two functions can be effectively integrated within a single DNF architecture (Schneegans, Spencer, & Schöner, in press); however, future work will be needed to probe whether the integrated architecture effectively captures both the data from change detection studies and the memory-based attentional biases examined here. 

The biasing effect of WM content is realized in the current model through the continuous bidirectional coupling between feature WM and feature attention. Recent experimental evidence indicates that there may be two forms of VWM: an active state that interacts with perceptual processing and an accessory or passive state that does not (Hollingworth & Hwang, 2013; Houtkamp & Roelfsema, 2006; Olivers et al., 2011). In Hollingworth and Hwang (2013), participants memorized two colors, one of which was subsequently cued as likely to be tested. In an intervening visual search task, only the cued color captured attention, as indicated by an increase in reaction time if that color was present as a distractor in the search display. Additionally, performance in the memory test was higher if the cued color was tested. We believe that such findings do not contradict the idea of continuous coupling between WM and attention. In fact, we propose that the combination of a multi-item WM representation with a selective (single item) representation for feature attention as implemented in the present model provides a potential mechanism for the different memory states. Critically, the accessory state is observed in experiments only when another WM item is in the active state. We propose that the active state is characterized by recruitment of the feature attention representation, such that mutually supportive regions of activation persist in both representations. The competitive interactions in the feature attention representations then suppress deployment of attention to other features and thereby prevent the other (passive) WM items from interacting with perceptual processing. The recruitment of feature attention would also stabilize the active WM memory item against random drift and decay of activation and thereby account for the increased memory performance for this item. It is yet to be tested whether this mechanism can indeed explain the different memory states, and further adjustments of the feature pathway and its parameters in the present model may be needed to account for the behavioral data. 

The combination of spatial and feature pathways with a shared low-level visual representation implements an architecture similar to several models of visual attention, in particular the neurodynamic models of Hamker (2003, 2005a, 2005b, 2006; also Fix et al., 2011). These models described dynamic interactions between spatial and feature representations to account for electrophysiological data on the time course of feature attention effects (Chelazzi, Duncan, Miller, & Desimone, 1998; Chelazzi, Miller, Duncan, & Desimone, 2001) and have shown how target features in VWM can guide spatial attention to produce visual search behavior. For these visual search tasks, the models can select object locations in a spatial representation through covert attention or as targets for a saccadic eye movement. Although these approaches employ population code representations for space and surface features that enable them to capture metric effects in a fashion analogous to the present architecture, they have not previously been used to investigate metric effects of feature WM on individual saccade amplitudes (though saccade latencies for different types of visual search tasks have been modeled in Hamker, 2005a), nor have any of these approaches addressed effects of perceptual processing and attentional selection on VWM representations. Nevertheless, we believe that the neural mechanisms underlying visual search are likely to be the same as those that produce the metric effects in saccade behavior and memory representations in the present study, as reflected by the analogous mechanisms in the computational models. 

One important conceptual aspect that we share with these previous approaches is how we conceive of the nature of visual attention. The deployment of attention is described as a continuous process that emerges from the interactions of different spatial and surface feature representations (as earlier proposed by Deco & Lee, 2002, 2004). The specific interaction patterns promote the selection of a single location and its associated surface features, which are represented more strongly at the expense of other locations and features. There is not, however, a strict requirement that attention has to be localized (it can be distributed at least transiently; compare Zirnsak, Beuth, & Hamker, 2011), and there is no discrete moment in time when an attentional selection takes place. This contrasts with many other models of visual search, where attentional selection occurs as a discrete, winner-takes-all selection in some form of spatial priority map (Navalpakkam & Itti, 2005; Wolfe, 1994). The resulting lack of distinct attentive and preattentive phases of processing in the present work is consistent with the empirical finding that effects of VWM occur even for simple saccades with latencies in the range of 80 to 130 ms (see, in particular, Hollingworth et al., 2013a). Such results suggest that the interactions between features held in VWM and the processing of visual objects take place at an early, sensory stage, influencing the first sweep of sensory processing following stimulus appearance. The resulting view implemented in the computational model is that the visual salience of a particular object is a joint property of the object's physical attributes, feature biases (e.g., match between those attributes and VWM content), and spatial biases (e.g., partial knowledge of the target location in the present task). 

Compared with extant models of visual search, the computational model presented here has several limitations. Some aspects of the implementation were intentionally simplified to reflect our focus on the details of low-level saccade planning and execution. In particular, we describe only a single spatial dimension because only horizontal saccade metrics were considered a relevant behavioral measure in the experiment. Moreover, the model describes interactions for only one surface feature dimension (color). In visual search tasks, the combination of different surface features is critical to capture the results of feature conjunction tasks and plays a central role in explaining the difference between serial and parallel search (Hamker, 2005a; Wolfe, 1994). However, a recent extension of the model presented here does introduce additional surface feature dimensions to address multifeature change detection tasks, and describes neural mechanisms for the parallel detection of simple feature changes and sequential detection of feature conjunction changes (Schneegans et al., in press). A further simplification in the present model (shared with the visual search models described here) is that it does not capture any increase in the complexity of visual features along the surface feature pathway. This is again driven by the focus on simple color effects in the experiment, although it ignores the generation of color representations from simple color opponency pairs at the earliest levels of perceptual processing. 

What the present work achieves compared with earlier approaches is to expand a dynamic explanation of visual attention from qualitative effects observed in visual search to metric effects in simple saccade planning and WM performance. The psychophysical experiments presented here and in previous related work (Hollingworth et al., 2013a, 2013b) provide a new method for investigating and quantifying the interactions between spatial and surface feature representations, and the results provide important constraints for modeling visual attention. The computational model demonstrates how VWM for surface features influences even the metric details of saccade planning and execution and how conversely the detailed content of VWM is affected by perceptual processing and attentional selection of visual objects. 

Supplementary Materials

pdf S1 

Acknowledgments

This study was supported by National Institutes of Health Grants R01-EY017356 and R01-MH062480, and by the German Federal Ministry of Education and Research within the National Network Computational Neuroscience - Bernstein Fokus: “Learning behavioral models: From human experiment to technical assistance”, grant FKZ 01GQ0951. 

Commercial relationships: none. 

Corresponding author: Sebastian Schneegans. 

Email: [email protected]. 

Address: Institut für Neuroinformatik, Ruhr-Universität Bochum, Bochum, Germany.