To bind or not to bind: Addressing the question of object representation in visual short-term memory

Abstract

Abstract

Abstract:

Abstract  Visualshort-term memory (VSTM) is a capacity limited resource, which is consistently estimated to hold about four visual items at a time. There is, however, debate in the literature about what constitutes an “item” and how resources are allocated within VSTM. Some research suggests information is stored in VSTM as discrete objects; however, there is also evidence suggesting that within-object features alter VSTM performance. The present study addresses the question of whether VSTM load effects reflect the number of discrete objects and/or the number of within-object features. An electrophysiological correlate of VSTM—the contralateral delay activity (CDA)—was measured while participants performed a lateralized change-detection task, in which to-be-remembered items varied in the number of features and locations. Each trial contained either a solitary simple feature (shape, color, or orientation) or one of two multifeature arrays: three features presented at three separate locations or three features bound at one location. While presenting multiple features—regardless of whether they are at discrete locations or bound within a single object—resulted in greater CDA amplitude relative to a solitary feature, there was a dissociation in the distribution of activity between the two multifeature conditions, such that the CDA at site P1/P2 was sensitive to the number of discrete objects, while activity at P7/P8 was most enhanced when multiple features were bound in one object. The findings demonstrate the inhomogeneity of the CDA and suggest this electrophysiological marker may reflect both discrete object individuation/separation and flexible feature-feature binding in VSTM.

Introduction

Visual short-term memory (VSTM), or the ability to maintain visual information in the mind after the objects have been removed, is capacity-limited. Across individuals, VSTM is estimated to hold roughly three to four items at a time (Cowan, 2000; Irwin & Andrews, 1996; Luck & Vogel, 1997; Todd & Marois, 2004; Vogel & Machizawa, 2004). While there has been some debate about VSTM capacity (see Oberauer, 2002), the more fundamental issues may concern what a unit of information in VSTM actually is and what constitutes VSTM “load.” According to the fixed slot model of VSTM, information or items are stored as discrete objects, regardless of object complexity (number of features within an object), and each object may be held or “slotted” into one of a fixed number of spaces or “placeholders” (Awh, Barton, & Vogel, 2007; Luck & Vogel, 1997; Zhang & Luck, 2008). Alternatively, the flexible resource model proposes that VSTM resources are distributed in a continuous manner, with greater feature complexity consuming more resources, resulting in a trade-off between object resolution and the number of distinct objects represented (Alvarez & Cavanagh, 2004; Bunge, Klingberg, Jacobsen, & Gabrieli, 2000). A great deal of research has been generated around this debate, with empirical work bolstering both models. Recent work on object files may inform the debate regarding the nature of VSTM representations given that literature's emphasis on the relationship between locations and the features that appear there. 

Object files and visual short-term memory

As originally conceived by Kahneman, Treisman, and Gibbs (1992), object files act as spatial indices, containing memory for the location of features. Location acts somewhat like a file folder, providing a node at which spatially specified features may be simultaneously yoked into a coherent visual object and individuated from features at other locations in the visual field (Treisman 1988; Treisman & Zhang, 2006). Location, as a perceptual glue, likely contributes to how features/objects are maintained in VSTM. The updating of this location information enables object perception to persist in the presence of visual disruption, such as motion, blinks, saccades, and occlusion. Given this role in object persistence, some have suggested that these bound feature/location tokens form the content of VSTM (Hollingworth & Rasmussen, 2010; Treisman & Zhang, 2006), a hypothesis consistent with a fixed slot model of VSTM. 

For example, Treisman and Zhang (2006) employed a typical VSTM change-detection paradigm, independently manipulating feature load (i.e., simple color vs. conjunction of shape and color) and probe location (i.e., same or novel location as the original memory array) to see whether memory for features was location-dependent. Their results suggest information may be initially encoded as object files in VSTM, as novel probe locations impaired memory, with a greater impact on memory for feature conjunctions relative to simple colors. Over the course of consolidation, however, representations of features become increasingly independent of the locations at which they were initially encoded, as the novel location cost was pronounced at shorter retention delays (100 and 900 ms) and dissipated at longer delays (3 and 6 secs). Logie, Brockmole, and Jaswal (2011) have also found evidence supporting this conclusion. Hollingworth and Rasmussen (2010) extended this work and suggested that the content of VSTM includes configural spatial relations that are flexible/dynamic. Taken together, the results of Treisman and Zhang (2006) and of Hollingworth and Rasmussen (2010) suggest that feature and location information may be encoded as bound object files, but over the course of consolidation and depending on task demands, feature-feature bindings and locations may be parsed and represented separately in VSTM. 

Evidence supporting this connection between object files and VSTM can also be found in neuroimaging work, which suggests that VSTM is composed of two separate, though likely codependent, systems allocated to the representation and maintenance of location and feature information. Activation in the inferior intraparietal sulcus (iIPS) scales with the number of locations, required for coarse object individuation, while activation in the superior intraparietal sulcus (sIPS) and lateral occipital complex (LOC) scales with feature complexity, involved in more fine-grained object identification (Xu, 2007, 2008; Xu & Chun, 2006). Xu and Chun (2009) take these findings as evidence supporting a neural object-file theory of VSTM, which is a dual-system model of VSTM. This approach integrates both fixed and flexible resource models, such that one system initially individuates visual objects via spatial location, forming coarse resolution discrete placeholders (location-based object folders), and the other system is responsible for object identification, selecting/encoding within-object features (files stored in location-based folders). 

Despite behavioral and neuroimaging evidence in the object file literature of distinct representations of feature-feature bindings and location information in VSTM, there remain contradictory results and debate in the electrophysiological literature about how information is represented, maintained, and constrained in VSTM. The present study aims to further advance this work on object files and VSTM by exploring how the spatial configuration of features—either spatially separated or bound together at one location—modulates an established electrophysiological signature of VSTM. 

Electrophysiological correlate of visual short-term memory

The contralateral delay activity (CDA), a sustained negativity that is maximal over posterior electrode sites contralateral to the attended side of a visual change-detection display, increases in amplitude with the number of to-be-remembered items and asymptotes at the same point at which behavioral estimates of VSTM capacity peak (e.g., Vogel, McCollough, & Machizawa, 2005). 

There is behavioral evidence that VSTM capacity is affected by visual load, such that fewer items can be maintained when those items have greater feature complexity (Alvarez & Cavanagh, 2004). While it has been argued that greater decision load associated with detecting relatively small changes at probe may account for these results (Awh et al., 2007), differences in decision load do not explain the load effects of complex objects on the IPS (Xu, 2007, 2008; Xu & Chun, 2006) or the CDA, as activity was recorded during the delay period prior to decision/response. The CDA is taken largely to reflect maintenance in VSTM; however, Awh and Jonides (2001) have pointed out overlap between spatial attention and VSTM, thus it is possible the load effects seen in the CDA may reflect the act of spreading attention across multiple objects. Whether the CDA may also reflect the act of binding multiple features within an object remains an open question. 

Despite this evidence of separate storage of feature and location information in the parietal lobe (Xu, 2007, 2008; Xu & Chun, 2006, 2009), most studies of the CDA have used simple feature change-detection tasks (i.e., color), and few have independently manipulated both the number of locations and the number of features, which no doubt has contributed to the ongoing contention regarding the nature of storage in VSTM. Woodman and Vogel (2008) used multifeature objects (colored bars rotated to different orientations) to show that participants could selectively upload only the relevant feature of such objects. The CDA, measured at a single electrode (PO1/PO2), was modulated primarily by the number of items retained, with a secondary effect of greater amplitude for a feature conjunction than for a simple feature alone; however, this effect appears to have been driven by a load-independent influence of orientation on the CDA. An enhanced CDA for polygons, relative to simple color stimuli, has also been found (Gao et al., 2009; Luria, Sessa, Gotler, Jolicoeur, & Dell'Acqua, 2010). Given that orientation is essentially a spatial property that must be bound to an object, it is possible that the larger CDA amplitude associated with orientation may reflect a binding load on VSTM. Similarly, random polygons contain a grouping of features—curves and angles—which also arguably must be bound in a spatial configuration. 

Another attempt to characterize how feature complexity affects the CDA involves adjusting the relative contrast of single-feature items (i.e., colors). Ikkai, McCollough, and Vogel (2010) found that CDA amplitude at three posterior electrode pairs was affected only by the number of items to be remembered and not by whether the colors were of high or low contrast, leading to their conclusion that the CDA—and thus VSTM capacity—is not affected by perceptual load. Although this contrast manipulation succeeded in making the task of discriminating colors more difficult, decreasing contrast is not equivalent to increasing the number of features within an object. 

In a recent study, Luria and Vogel (2011) manipulated both the number of features and locations of those features (i.e., either spatially separated or bound together in a single object) across two experiments. They found that the CDA was significantly larger for multifeature objects relative to single color objects during the early phase of the CDA (400 ms–600 ms), but not in the late phase (750–1000 ms), providing some evidence that bound features place greater load demands on the CDA than single features alone. Furthermore, the CDA was larger for four colors at separate locations relative to two multifeature objects (four features in total), thus the authors concluded the CDA primarily scales with the number of discrete objects rather than the number of features within objects. 

This study, however, has a few limiting factors that leave this question of representation in VSTM open. For example, the smaller amplitude in the multifeature condition does not entail that there is no effect of binding at all on the CDA. It is quite possible CDA magnitude, in general, may be larger for maintenance of multiple discrete objects relative to those bound within a single object, as there may be an extra cost associated with spreading attention over space (Awh & Jonides, 2001), which may enhance CDA amplitude. Luria and Vogel's (2011) behavioral results are consistent with this notion, as there was a behavioral cost for two simple objects relative to a two-feature object in Experiment 1 and for four simple objects relative to two multifeature objects in Experiment 2. If an amplitude difference for within-object feature binding is more subtle than that for discrete objects, a more robust contrast may be required—i.e., between one and three features rather than one and two features. 

Luria and Vogel did not provide a satisfying account of the significantly enhanced CDA amplitude for multifeature objects during the early window and were perhaps premature in dismissing this finding, which we think calls for further exploration. Most studies involving the CDA use memory arrays with greater than two items (see Ikkai et al., 2010; Vogel et al., 2005; Woodman & Vogel, 2008). Visual inspection of the event-related potential (ERP) plots in one of the original papers on the CDA (Vogel & Machizawa, 2004) shows an initial amplitude difference between one and two items early in retention (400 ms–600 ms); but by about 700 ms, this difference disappears, suggesting that, over time, the CDA distinguishes less even between one and two discrete items.Thus, finding an enhanced CDA response to a multifeature item during the early window is consistent with this early work showing load effects on the CDA, suggesting the CDA does reflect within-object feature complexity. 

Another potential limiting factor in the Luria and Vogel (2011) study, as well as other studies (i.e., Woodman & Vogel, 2008), is the restriction of CDA analyses to a single electrode pair—i.e., the topographical peak of the CDA. Following from Xu and Chun's work, suggesting a dual-system model of VSTM, it is possible different electrodes may be more or less sensitive to the number of discrete objects compared to the number of within-object features. Though electrode interactions cannot confirm or counter claims about whether there are separate neural generators of the CDA, they may reveal dissociations in how the CDA, as an index of VSTM maintenance, responds to feature context—i.e., features at separate locations versus bound within a single object. 

The present study builds on this previous work, aiming to establish whether this signature of VSTM is modulated by the number of features within objects or by the number of features at separate locations. In an attempt to circumvent the possibility of missing small amplitude effects in the CDA related to within-object feature binding, we contrasted one and three features instead of one and two features, which could appear at either one or three distinct locations. Participants were required to maintain all the information presented on every trial. The CDA was measured as an index of active representations in VSTM. We compared CDA amplitude across three conditions: a low-load or single feature condition in which a solitary feature was presented at one location and two higher-load or multifeature conditions in which either three features were presented bound at one location or three features were presented at three separate locations. This design allowed a comparison between the effect of feature binding and object individuation on the CDA. In addition to standard amplitude analyses, which included separate analyses of the CDA across multiple posterior channels and the topographical peak of the CDA (PO7/PO8), the topographical distribution of the CDA was examined to address the possibility that the two multifeature manipulations (bound and separated features) may result in dissociable load effects on the CDA at distinct electrode sites. Given evidence that there may be distinct VSTM systems dedicated to within-object feature binding and object individuation, we expected to find a double dissociation between feature context and electrode site, such that the CDA at some electrode sites would be enhanced when presented with multiple features within an object, but would treat multiple features at distinct locations like solitary features, reflecting sites sensitive to within-object feature binding. Conversely, the CDA at other sites would be most enhanced when presented with multiple features at distinct locations, treating single multifeatured objects like single solitary feature objects, reflecting a process of object individuation/separation. Furthermore, examining electrode sites may address questions concerning the influence of distributing spatial attention across locations on the CDA (i.e., Awh & Jonides, 2001). More specifically, if the CDA is primarily affected by attentional demands associated with spreading attention over space in order to individuate and encode distinct items, then it should scale with the number of discrete items, regardless of within-object features. Furthermore, this effect should be fairly consistent across electrode sites where the CDA is typically found. A double dissociation, however, where some electrodes scale with binding while others scale with the number of discrete items would help rule out a strict attentional account of CDA behavior. 

Methods

Participants

Thirty-one young adults were recruited from the University of Toronto community. Of these, data sets from 22 participants met selection criteria for inclusion in analyses (see Results). These participants (13 females, all right-handed; age range: 18–31 years; mean: 23 years) had normal or corrected-to-normal vision and gave informed consent in accordance with the University of Toronto Ethics Review Board. 

Apparatus

The experiment was programmed and displayed using Presentation software (Version 11.0, www.neuro-bs.com) running on a desktop PC with a ViewSonic 21-inch monitor (1,600 × 1,200 resolution, 60 Hz refresh rate). The viewing distance was 57 cm, and participants made responses with their right hand on the left and down arrow keys of a standard keyboard. Continuous, unreferenced EEG was recorded at a sampling rate of 512 Hz using a BioSemi ActiveTwo system with 64 scalp electrodes in standard 10–20 placement with additional electrodes at each mastoid, the outer canthus of each eye, and below each eye. 

Stimuli and procedure

All stimuli were presented against a white background with a black, central fixation cross (subtending 1°) visible throughout each trial. Left- or right-pointing black arrows (2° wide) were presented 2° above the central fixation cross to direct attention to either side of the bilateral display. Memory and probe stimuli were one of four colors (red, blue, green, or yellow), one of four unfamiliar grey shapes, one of four blocks of oriented grey lines (0°, 45°, 95°, or 135°), or combinations of one feature from each category (adapted from stimuli used by Barense et al., 2005; see Figure 1 for a sample). Stimuli were all scaled to 2.43° by 1.69° and were presented randomly in 4° by 8° rectangular areas centered 5° to the left and right of the central fixation cross, with the constraint that potential locations were all at least 2° apart. 

Figure 1

 

Samples of single features and how they combine to form a multifeature object.

Each trial was initiated when the participant pressed the spacebar, after which the fixation cross appeared alone for 300 ms, followed by the presentation of a left or right arrow for 200 ms that directed the participant to attend only that side for the duration of the trial (while maintaining central fixation, see Figure 2). After a variable delay from 300 to 400 ms, a memory array appeared containing either one single-feature, one multifeature, or three different single-feature items at random locations on each side of the display (see conditions in the following text). The memory array was visible for 200 ms, followed by a delay of 1000 ms, after which the probe appeared for 2000 ms. Participants were to indicate whether any single object or feature on the attended side for that trial had changed from the memory array to the probe, using the index finger for matches and the middle finger for nonmatches. Each intertrial interval (ITI) was self-paced, lasting however long the participant needed to blink and reset for the next trial. 

Figure 2

 

Schematic illustrations of change-detection trials for (a) Solitary feature trials in which a single feature was to be maintained, (b) Separated feature trials in which single features at three locations were to be maintained, and (c) Bound feature trials in which three features were to be maintained at one location. (Note: The single probe version of the task is shown here; for the full probe version, the Sep condition probe contained three features.)

On Solitary object (Solo) trials, the memory array contained one single-feature item chosen from the same feature category on each side (i.e., one color on each side, or one shape on each side, or one oriented grating/texture on each side, with an equal likelihood of each feature class appearing). The probe was then a single-feature item from the same feature class as the memory array. On Bound object (Bnd) trials, the memory array contained multifeatures bound within a single item, containing a feature from each category (i.e., one particular shape containing a particular oriented texture in a particular color) presented on each side of the screen. The probe was also a multifeature object that was equally likely to be a match on all three features or a nonmatch on any one of the three features (e.g., a single change in shape while the color and oriented texture of the object remained the same). On Separate object (Sep) trials, three single-features, one from each category (i.e., one color, one shape, and one orientation), were presented at separate locations, resulting in three distinct items on each side of the screen. Two versions of the task were run with different participants, in which the Sep probe was either a single probe or full probe, as it has been shown elsewhere that the probe type can impact feature memory (Treisman & Zhang, 2006). Twenty participants (13 included in analyses) were tested with a single probe consisting of a single-feature item on each side of the display, leaving no ambiguity regarding which item from the memory array was to be compared (Single probe version). The remaining 11 participants (nine included in analyses) were tested using a full probe array in which single-feature stimuli were presented at all locations occupied in the memory array, requiring participants to identify which, if any, of the items were nonmatching (Full probe version). In either case, features from the same class appeared at specific locations in the memory array and probe (e.g., a shape might change to another shape but not to a color). 

In both probe versions, there were no constraints regarding overlap between features on each side (e.g., a blue stimulus may or may not appear on both sides). The match/nonmatch trials were counterbalanced on each side and fully crossed with the side that had to be attended, such that a probe was equally likely to be a match or nonmatch, and the decision for the attended side was completely independent of the unattended side. A total of 576 trials were presented, with 192 trials per feature context condition (i.e., Solo, Bnd, Sep). All trial conditions were intermixed and presented in random order. 

After obtaining informed consent and preparing for EEG recording, participants viewed all possible stimuli and were informed of the various task conditions. They were instructed to attend only the side indicated by the arrow on each trial while keeping their eyes fixated on the cross at the center of the display. Participants were told that if items changed after the delay, they would differ by only one feature. The emphasis was placed on judging as accurately as possible whether the probe was the same or different, and participants were asked to make a “best guess” on trials for which they were uncertain of the correct response. 

ERP processing

EEG datasets were converted into EEProbe format (version 3.3.118) using PolyRex (Kayser, 2003). Each individual data set was filtered using a finite impulse response (FIR) filter high-passed at 0.01 Hz and low-passed at 30 Hz. The continuous data were then re-referenced to the average of the two mastoids. Additionally, we computed lateral eye movements by re-referencing the right outer eye to the left outer eye. Rejection markers were then generated over all eye channels, frontopolar channels FP1 and FP2, and all posterior channels contributing to analyses to guarantee exclusion of data that exceeded a standard deviation of 20 μv within a sliding window of 200 ms. Each continuous data set was averaged to generate individual ERPs for 1400-ms windows beginning 200 ms prior to and ending 1200 ms after the onset of the memory array. The 200 ms prior to the memory array onset was used as a baseline, and the rejection window spanned from 600 ms prior to 1000 ms after the memory array onset to ensure that trials with artifacts (such as saccades) following the arrow were rejected. Average waveforms for each individual were imported to Matlab (version 7.0.4.352) for analyses using custom scripts, including generation of difference waves for contralateral-ipsilateral electrode pairs. 

Results

Exclusions

Four participants were excluded from analyses (two within the single probe version and two from the full probe) due to performance that was more than two standard deviations from the group mean. Furthermore, full data sets were excluded from analyses if more than 35% of trials were rejected due to excessive EEG artifacts, which resulted in the exclusion of an additional five data sets (four and one for single and full probe versions, respectively). Although all trials contributed to behavioral analyses within the 22 remaining data sets (13 and 9 in the single and full probe versions, respectively), an average of 9.8% of trials were rejected from ERP analyses due to EEG artifacts (no difference between probe versions, p > 0.5), leaving approximately 346 trials per feature context condition on which CDA analyses were performed. 

Behavior

Change-detection performance was measured with respect to accuracy, defined as the percentage of both matching and nonmatching trials with correct responses (hits and correct rejections) on trials with correct responses. Individual participants' measures of accuracy were submitted to repeated-measures ANOVAs with the single within-subject factor of feature context—Solitary (Solo—single feature, single location) versus Separate (Sep—three features each at a separate location) versus Bound (Bnd—three features bound together at a single location)—and a between-subjects factor of probe version (Single vs. Full). Feature context had a significant effect on accuracy (F[2, 40] = 148.54, MSE = 11.18, p < 0.001). Bonferroni-corrected pairwise comparisons showed an accuracy advantage for Solo relative to Sep (p < 0.001) and Bnd (p < 0.001), and for Bnd relative to Sep (p < 0.001). The probe version had neither a main effect on accuracy (F[1, 20] = 0.891, MSE = 87.4, p = 0.89) nor did it interact with feature context (F[2, 40] = 2.47, p = 0.1). Figure 3 displays the group results for these measures collapsed across probe version, as probe version did not significantly impact performance. The lack of any effect of probe type on accuracy served to justify collapsing the ERP analyses across both probe versions. Further, as one might expect, given that CDA measurements were confined to the delay period prior to the appearance of the probe, when we compared ERP data across the two probe versions, no main effects or interactions with probe versions were found (see following results). 

Figure 3

 

Group mean accuracy (percent correct) with error bars representing standard error of the mean (SEM).

Contralateral delay activity

Overview of key analyses steps

The CDA was defined as the mean amplitude of the contralateral-ipsilateral difference wave from 400 to 1000 ms after the onset of the memory array, generated separately for each feature context. We first explored the topographical peak of the CDA at sites PO7/PO8 (as commonly done in CDA studies; e.g., Gao et al., 2009; Ikkai et al., 2010; Luria et al., 2010; Vogel et al., 2005) in order to address the question of whether the CDA at PO7/PO8 showed load effects associated with Sep or Bnd trials, relative to Solo trials. Following from concerns regarding stimulus imbalances in the Solo condition, another ANOVA was used to address the question of whether there were CDA amplitude differences at PO7/PO8 between Solo features (orientation, shape, color). Next, we explored a cluster of six posterior electrode sites, also a common method used to measure the CDA. We started with an omnibus analysis across six posterior electrodes in order to address the question of whether the CDA at these six electrode pairs showed a pattern similar to that at PO7/PO8. The ANOVA included two within-subject factors—electrode (six pairs) and feature context (Bnd, Sep, Solo)—as well as a between subject factor of probe version. This omnibus ANOVA revealed a significant feature context by electrode interaction. We then zoomed in on the interaction by generating a difference wave between the Sep and Bnd conditions, eliminating any influence the stimulus imbalance in the Solo condition may have had and isolating the effect of feature binding from feature separation on CDA activity. This was then followed by a test of simple effects, where the Solo condition was reintroduced; however, this analysis was restricted to only those electrode sites that did not show stimulus imbalances in the Solo condition. 

Topographical peak of the CDA—PO7/PO8

The topographical peak of the CDA (electrode pair PO7/PO8; see Figure 4) showed an effect of feature context that only trended toward significance (F[2, 40] = 2.79, MSE = 1.19, p = 0.07). Bonferroni-corrected pairwise comparisons revealed a single difference: larger CDA for Bnd relative to Solo (p = 0.05); however, the difference between Sep and Solo did not reach significance (p = 0.28) or Sep (p > 0.9) and Bnd (p > 0.9). There was no interaction with probe version (F[2, 40] = 0.41, p = 0.67). 

Figure 4

 

(a) Grand average waveforms. The grey bars represent the duration of the memory array (from 0–200 ms) and the onset of the probe (at 1200 ms), respectively, and (b) group mean amplitudes at the topographical peak of the CDA (PO7/PO8). There was no significant difference between Sep and Bnd trials. Error bars represent SEM.

CDA amplitude differences at PO7/PO8 between Solo features

Previous work has shown that the CDA is enhanced for oriented bars (Woodman & Vogel, 2008) and random polygons (Luria et al., 2010), relative to simple color stimuli. It is possible that the Solo trials in the present study were attenuated due to an imbalance in the number of trials that contained polygons or oriented texture stimuli. Each trial in the Bnd and Sep conditions contained all three feature types, whereas 2/3 of the trials in the Solo condition contained either a polygon or oriented texture, which may have contributed to the smaller CDA in this condition. To address concerns about whether the CDA varied across the different Solo conditions (color, shape, orientation), a repeated-measures ANOVA, with the single within-subject factor of Solo conditions, was conducted at PO7/PO8, the topographical peak of the CDA. Here, we collapsed across probe, as no interaction with probe type was found in the initial analysis. This analysis revealed a marginal effect (F[3, 42] = 2.86, MSE = 2.76, p = 0.07). Pairwise comparisons showed this effect to be driven by a single difference—the CDA for the Solo-oriented stimuli was slightly larger than the Solo-color stimuli (p = 0.03), with the intermediate Solo-shape not differing from either the oriented (p > 0.9) or color stimuli (p = 0.5). Thus, it is possible that the lower amplitude for the Solo condition may be, at least in part, due to one-third of the trials containing only a simple color, while the multifeature conditions contain a polygon and oriented stimuli on every trial. This concern is addressed in the following analyses of the six posterior electrode sites. 

Omnibus analysis across six posterior electrodes

The effect of feature context was examined by including six posterior site pairs that typically show a CDA in our analyses (P1/P2, P3/P4, P5/P6, P7/P8, PO3/PO4, PO7/PO8; see Figure 5 for group average waveforms and mean amplitudes, collapsed across six electrode pairs). The average difference waves were then submitted to a repeated-measures ANOVA with two within-subject factors of feature context and electrode, and one between-subject factor of probe version. This analysis showed an effect of feature context (F[2, 40] = 6.95, MSE = 1.46, p = 0.003). Bonferroni-corrected pairwise comparisons revealed a pattern similar to that found at the topographical peak (PO7/PO8). Relative to Solo trials, the CDA was significantly larger for Bnd (p < 0.001) and marginally larger for Sep (p > 0.06), and Sep and Bnd did not differ (p > 0.9). Probe version showed no main effect on the CDA (F[1, 20] = 0.75, MSE = 10.69, p = 0.4) nor did it interact with feature context (F[2, 40] = 0.82, MSE = 1.2, p = 0.45) or electrode (F[5, 100] = 1.37, p = 0.24). Electrode was a significant predictor of CDA amplitude (F[3.1, 100] = 9.64, MSE = 16.1, p < 0.001).¹ Furthermore, a significant interaction between electrode and feature context was present (F[8.1, 162.8] = 1.97, MSE = 2.13, p = 0.05). Given previous work suggesting two distinct systems in VSTM (one individuating features and the other binding features; Xu & Chun, 2006, 2009), answering questions about the sensitivity of the CDA to object separation and feature binding may depend on the electrode considered. Though interactions with electrode site cannot elucidate the neural sources underlying electrical activity recorded at the scalp (Luck, 2005; Urbach & Kutas, 2002), a double dissociation in CDA responsiveness to feature binding versus separation at different electrode sites where the CDA is typically found would bring into question the claim that the CDA is only sensitive to the number of discrete items in an array. Given that CDA amplitude was significantly smaller for Solo-color trials at PO7/PO8, before testing the simple effects of this interaction between feature context and electrode, however, we tested whether simple feature effects may have spuriously contributed to this interaction. If this is the case, further analyses of this interaction should be restricted to the difference in CDA amplitude between Sep and Bnd conditions across electrode cites. 

Figure 5

 

(a) Grand average waveforms collapsed across six posterior sites (P1/P2, P3/P4, P5/P6, P7/P8, PO3/PO4, PO7/PO8) for each of three conditions. The grey bars represent the duration of the memory array (from 0–200 ms) and the onset of the probe (at 1200 ms), respectively. (b) Group mean amplitudes of the difference waves depicted in (a) show significant differences between Solo and Bnd and Sep conditions and marginal difference between Bnd and Sep. The error bars represent SEM.

Differences in Solo feature effects across six posterior electrodes

The average across the six posterior electrodes was submitted to a repeated-measures ANOVA with a within-subject factor of Solo condition, and we again collapsed across probe version. This analysis showed a main effect of feature type (F[3, 42] = 4.22, MSE = 0.544, p = 0.02), such that Solo-color was marginally smaller than Solo-shape (p = 0.06) and significantly smaller than Solo-oriented (p = 0.03), whereas Solo-oriented and Solo-shape did not differ from each other. Because of this discrepancy in CDA responsiveness to simple features, we cannot be sure whether amplitude differences between Solo and the two multifeature conditions reflect real load differences or whether they reflect differences in stimulus properties. In the following analyses, we use a subtraction method to isolate the key differences in CDA activity between Bnd and Sep conditions before following up on the original electrode by feature context interaction with a simple effects test. This step identifies electrodes that show a critical dissociation between CDA responsiveness to feature context, i.e., bound versus separated at discrete locations, independent of Solo condition. 

Isolating the effect of feature binding from feature separation

The comparison between Bnd and Sep trials was of greatest relevance to the primary question behind this study, namely whether the CDA is sensitive to both the number of discrete locations and/or the number of features bound within a single object. Our analyses of the CDA thus far suggest this ERP does not differ significantly between discrete and bound features at either PO7/PO8 or when averaging across six posterior electrodes; however, given the significant interaction reported in the “Omnibus analysis across six posterior electrodes” section, here we explore the possibility that the CDA may be sensitive to both feature context manipulations, depending on the electrode observed. Thus, the following analyses focus specifically on the Bnd and Sep conditions. 

We explored the interaction between feature context and electrode by first isolating activity unique to Bnd and Sep trials through subtracting activity on Bnd trials from that on Sep trials at each of the six electrode pairs. These condition-based difference waves were submitted to repeated-measures ANOVA with a within-subject factor of electrode. Given that none of the analyses thus far showed any effect of probe, we again collapsed across probe version. Electrode was a significant predictor of location-based differences in CDA amplitude (F[5, 105] = 3.24, MSE = 1.76, p = 0.009). Bonferroni-corrected pairwise comparisons revealed that this was driven by a single difference, between electrode P1/P2 and P7/P8 (p = 0.05). Figure 6 illustrates this difference with a positive difference wave at site P7/P8 reflecting a larger CDA in response to bound features and a negative difference wave at site P1/P2 reflecting a larger CDA for separate features or discrete objects. 

Figure 6

 

(a) Grand average waveforms for P1/P2 and P7/P8, (b) group mean CDA amplitudes for P1/P2 and P7/P8, and (c) isolated activity associated with larger CDA for Sep relative to Bnd trials at P1/P2 (red) and for Bnd relative to Sep trials at P7/P8 (blue). The grey bars represent the duration of the memory array (from 0–200 ms) and the onset of the probe (at 1200 ms), respectively. Error bars represent SEM.

Solo feature effects at sites P1/P2 and P7/P8

The dissociation between P1/P2 and P7/P8 shows that the CDA responds differently to Sep and Bnd trials, depending on the electrode observed, revealing the source of the original electrode by feature context interaction. In order to make a strong case for the effects at P1/P2 and P7/P8 reflecting load effects, however, it would be helpful to establish that the Sep effect at P1/P2 and the Bnd effect at P7/P8 are in fact larger in amplitude relative to a single simple feature condition. If there is no difference in CDA amplitude across the different Solo features at these two electrode sites, this would warrant reintroducing this condition in a final set of analysis, exploring the simple effects of the electrode by feature context interaction revealed in the first ANOVA. Thus Solo feature conditions were submitted to a repeated-measures ANOVA, containing with-subject factors of electrode (P1/P2 and P7/P8) and Solo feature (orientation, shape, and color). There was no significant difference between Solo features, F(2, 42) = 2.08, MSE = 2.17, p = 0.14. There was no effect of electrode, F(1, 21) = 1.63, MSE = 5.71, p = 0.22, nor was there a significant interaction between Solo feature and electrode, F(2, 42) = 2.67, MSE = 1.88, p = 0.1, which justified using the Solo condition as a load contrast at these two electrode sites. 

Load effects at P1/P2 and P7/P8

In this final analysis, we use pairwise t tests to explore simple effects and the specific hypothesis that the CDA at P1/P2 scales with the number of discrete objects, and thus does not distinguish between Bnd and Solo trials, while the CDA at P7/P8 is modulated by within-object complexity, and thus equates Sep trials with Solo trials. The results support this hypothesis. At site P1/P2, Solo was significantly smaller than Sep (t[21] = 3.33, SEM = 0.26, p = 0.003). However, Solo and Bnd did not differ significantly (t[21] = 1.14, SEM = 0.25, p = 0.28), whereas at site P7/P8, the reverse was true with significantly smaller CDA for Solo trials relative to Bnd (t[21] = 2.79, SEM = 0.44, p < 0.01), but no difference between Solo and Sep (t[21] = 0.71, SEM = 0.34, p = 0.48). 

CDA and Sep accuracy cost

The relationship between CDA amplitude and performance usually emerges when set size is manipulated and individual capacity is exhausted. Given the dissociation in CDA response at P7/P8 and P1/P2, it was conceivable that activity at these sites would predict individual difference in accuracy on this task. We used Pearson's correlation to determine whether CDA amplitude at P1/P2 in the Sep condition and P7/P8 in the Bnd condition were associated with accuracy in Sep and Bnd conditions, respectively. We also looked at the relationship between CDA amplitude at these two electrode pairs and three different accuracy cost calculations: (a) the Sep condition relative to the Bnd (Sep accuracy minus Bnd accuracy), (b) Sep relative to Solo (Sep minus Solo), and (c) Bnd relative to Solo (Bnd minus Solo). All correlations were small and nonsignificant (p > 0.1) except for one significant positive correlation between the CDA at site P1/P2 in the Sep condition and the behavioral cost associated with Sep relative to Bnd (r = 0.649, p = 0.001), and a marginal positive relationship between the CDA at P1/P2 in the Sep condition and the cost associated with Sep relative to Solo (r = 0.38, p = 0.08). In other words, these two positive correlations indicate that a larger CDA amplitude in the Sep condition may reflect higher demands on VSTM resources as attention is distributed across several locations, relative to a single location, providing an explanation of the behavioral cost associated with the Sep condition (Figure 7). 

Figure 7

 

Correlation plot reflecting CDA amplitude at P1/P2 on Sep trials and the (a) accuracy cost for Sep relative to Bnd trials (difference in performance: accuracy for Sep trials minus accuracy for Bnd trials [r = 0.649, p = 0.01]) and (b) accuracy cost for Sep relative to Solo trials (accuracy for Sep trials minus accuracy for Solo trials [r = 0.38, p = 0.08]).

Discussion

In summary, the behavioral and electrophysiological measures of VSTM performance showed evidence of increased processing requirements when three to-be-remembered features were presented either bound at one location (Bnd) or spread across three locations (Sep) relative to a solitary simple feature presented at one location (Solo) with an additional performance cost for Sep relative to Bnd trials. Whereas the topographical peak of the CDA showed little distinction between single and multifeature loads, a clear difference in amplitude was observed between the single-feature and the two multifeature conditions when collapsing across six posterior electrodes. An interaction between electrode and feature context was explored by first isolating the unique contribution of each of the two multifeature conditions on CDA amplitude at each electrode site. This exposed a clear dissociation between the two multifeature conditions at two electrode pairs, such that the CDA at electrode pair P1/P2 was most enhanced when features were spatially distinct, whereas the CDA at P7/P8 was selectively amplified when multiple features were bound at one location. These results may be taken as evidence that both object individuation and identification or binding have amplitude effects on the CDA and thus VSTM, bolstering a dual-system model of VSTM (e.g., Xu & Chun, 2006, 2009). 

Behavioral cost for three individuated and three bound features

Overall accuracy was lower for Sep features and Bnd features relative to Solo features, suggesting that both multifeature manipulations placed increased processing demands on VSTM. Accuracy was also significantly lower for Sep trials relative to Bnd trials. The finding of reduced accuracy for multiple objects relative to a single simple or single multifeature object is somewhat consistent with the fixed slot model, reflecting greater load effects on the CDA with an increase in number of bound object files. The performance cost for Bnd relative to Solo trials may reflect increased load on VSTM resources associated with feature binding, which on first review may appear to lend support to a flexible resource or dual-system model of VSTM. Previous work, however, has shown that multifeature stimuli result in a greater decision load, which negatively impacts performance (Awh et al., 2007), and this may underlie the observed behavioral cost for Bnd relative to Solo trials. Furthermore, the behavioral results alone cannot rule out the possible contribution that differences in perceptual load may have contributed to the overall cost for multifeature trials relative to single feature trials. The ERP results, however, bring into question the plausibility of both this interpretation and one suggesting multiple bound features are treated like a single feature object in VSTM with greater load effects only reflecting the number of items at discrete/separate locations. 

Divergent effects of feature context on CDA

Analysis of the topographical peak of the CDA (PO7/PO8) showed an effect of feature context that only trended toward significance with a larger CDA for Bnd relative to Solo trials, while the intermediate amplitude for Sep trials did not differ from either of the other two conditions. Collapsing across six posterior electrodes revealed a slightly different pattern with a significant difference in CDA amplitude as a function of the number of features driven by an enhanced CDA for both multifeature conditions relative to the solitary feature condition. These differences in amplitude across posterior sites are consistent with the pattern in accuracy across the different conditions and suggest that CDA amplitude—and thus VSTM—may represent not only the storage of discrete items, but also the number of features bound within an object. Furthermore, feature context interacted with electrode, which motivated follow-up analyses. Due to stimulus imbalances in the Solo condition relative to Bnd and Sep conditions, however, the initial analyses of the electrode interaction were restricted to the critical difference between the two multifeature conditions in order to ensure that stimulus property imbalances were not the main driving force behind this interaction. 

At each electrode site, activity on Bnd trials was subtracted from Sep trials, leaving behind residual activity that can be attributed to the manipulation of feature context. This subtraction has the advantage of eliminating any activity related to properties these two conditions share and enabling the key difference between these two conditions to be isolated, that is, feature separation versus binding in VSTM. Moreover, this subtraction also helped to abate concerns regarding the possible confound resulting from feature imbalances between the Solo condition and the two multifeature conditions, as Solo trials were excluded from this analysis. Analysis of this isolated CDA activity showed significant dissociable effects at two electrode pairs with P1/P2 responding preferentially to multiple features at separate locations, whereas P7/P8 was sensitive to multiple features bound at a single location. Further, after first confirming there were no differences in CDA amplitude across the three Solo feature conditions at these two electrode sites, we looked specifically at load effects (i.e., difference between Solo and Bnd, and Solo and Sep) and found evidence of a consistent dissociation in the CDA at P1/P2 and P7/P8. That is, at P1/P2, the CDA did not distinguish between Solo and Bnd trials, but was larger for Sep than Solo trials. In other words, the CDA at this electrode pair was insensitive to the number of features bound in a single object, but rather scaled in amplitude with the number of spatially distinct objects/features. 

Conversely, at P7/P8, the CDA did not distinguish between Solo and Sep trials, but was larger for Bnd than Solo trials. In other words, the CDA at this electrode site was insensitive to the number of discrete objects, but rather showed enhanced amplitude when multiple features were bound at one location. This double dissociation not only establishes load effects for both feature binding and feature separation in the CDA, depending on the electrode observed, but undermines the plausibility that differences in perceptual load gave rise to the behavioral and main CDA effects reported here as, at each of these two electrode pairs, one of the conditions with higher perceptual load (Sep or Bnd) was equated with the condition with lower perceptual load (Solo). Further, the difference in CDA amplitude between Solo and both multifeature conditions cannot be accounted for by differences in decision or comparison load, as the CDA reflects capacity limitations during maintenance and is not susceptible to errors resulting from comparison difficulty or decision load at the end of the trial (see Awh et al., 2007; Ikkai et al., 2010; Luria et al., 2010). 

The present study extends previous work using the CDA to explore representation in VSTM, such as that of Luria and Vogel (2011), in several ways. First, in the present study, the magnitude of the feature load manipulation was larger than that used by Luria and Vogel—i.e., one compared to three features as opposed to contrasting one and two features. This may have been a limiting factor in the previous study, as the contrast between one and two items is a rather small load manipulation, which may not be enough to dissociate binding load effects from discrete object load effects. This change in the magnitude of our load manipulation showed an effect of binding on the CDA comparable to the effect of individuating discrete objects at electrode PO7/PO8 and across six posterior electrodes, which is somewhat consistent with the results of Experiment 2 in Luria and Vogel (2011), where they report an amplified CDA at OL/OR (comparable to PO7/PO8) for the conjunction of two squares compared to one square during the first half of the retention interval. That the CDA is sensitive to binding feature load to the same degree as separating feature load in our data, unlike what Luria and Vogel report, might be related to the magnitude of the binding effect in the CDA. The behavioral data in both the present study and Luria and Vogel's study is consistent with the notion of a memory advantage for bound features, which arguably may be associated with a smaller effect on the CDA. In our study, we used not only more features, but the features themselves were arguably more complex than the simple squares used in the Luria and Vogel study, which may have contributed to the larger feature binding effect in the CDA. Memory for random polygon shapes and for oriented gratings arguably requires greater discrimination of within-object features and the spatial configuration of those features than is required for simple colors, leading to a greater binding load in our study. Further, if the CDA is sensitive to complexity/binding as our data suggest, this may explain why the CDA is amplified for these two features, relative to simple squares (see Gao et al., 2009; Luria et al., 2010; Woodman & Vogel, 2008). Further still, in the present study, we explored electrodes not included in the Luria and Vogel study and performed a subtraction between Bnd and Sep trials, which enabled the isolation of CDA activity unique to each of these manipulations, which in fact varied with electrode. 

Typically, relationships between behavioral performance and CDA emerge when set size is manipulated, which was not an element of this study; nonetheless, a correlation between behavior and CDA amplitude was found. Specifically, CDA amplitude at the site P1/P2 on Sep trials was predictive of the behavioral cost associated with Sep trials—a larger CDA was predictive of lower accuracy in the Sep condition relative to both Bnd and Solo conditions. This behavioral cost and larger CDA amplitude for Sep trials may reflect a greater demand on VSTM resources as a function of allocating attention to several discrete locations relative to consolidating resources at one location. This interpretation is consistent with evidence presented by Awh and Jonides (2001) suggesting attention plays a critical role in VSTM. Given that the CDA is taken to reflect the allocation of resources during the maintenance stage of VSTM, which scales with load, and that memory errors increase with greater load (see Vogel, Woodman, & Luck, 2001), the correlation between CDA amplitude in the Sep condition and behavior suggests individuating features at distinct locations place a greater load on VSTM resources than maintaining a single feature item or multiple features bound by a single location. Furthermore, the correlation draws an important link between the electrophysiological findings and behavioral outcomes in the present study, suggesting that the amplitude differences observed in the CDA reflect storage in VSTM. 

Electrophysiological evidence of neural object file theory

Due to the coarse spatial resolution of EEG and the challenges inherent to localizing ERP sources, the current results cannot be mapped onto the structures observed by Xu and Chun (2006)—iIPS associated with location-based object individuation, sIPS, and LOC associated with feature-based object identification. Nonetheless, their theoretical framework—namely their neural object file theory or dual-system model of VSTM—may provide a parsimonious account of the behavior of the CDA at electrode sites P1/P2 and P7/P8. The system indexed by the CDA at P1/P2 scaled with the number of discrete objects, irrespective of feature complexity, showed greatest amplitude for Sep trials. The CDA at this electrode pair may reflect the number and location of distinct objects (object files) and be limited by the number of discrete items that can be attended or individuated at a given time, constraining the upper-limit of VSTM capacity to about four items (see Cowan, 2000). This interpretation is consistent with work by Awh and Jonides (2001) that suggests locations are maintained in VSTM by allocating attention to them, and if attention is withdrawn, VSTM suffers. Conversely, the system indexed by the CDA at P7/P8 was enhanced for multiple features bound at a single location with no regard for number of discrete items and largest amplitude for Bnd trials. The process reflected at this site may be involved in maintaining fine-grained feature details and be limited by the process of distributing resources within individuated object files, determining the resolution of each representation, altering identification, and accounting for lower capacity estimates for complex stimuli relative to simple stimuli (Hollingworth & Rasmussen, 2010; Saiki, 2003; Treisman & Zhang, 2006). Relative to maintaining several discrete simple features and spreading attention over space, binding features at a single location may facilitate memory in basic change-detection tasks, an interpretation supported by the accuracy cost observed in the present study and for Bnd relative to Sep trials (a similar behavioral advantage for bound items found in Luria and Vogel, 2011). The CDA at P7/P8 appears to be specific to the binding process, which is a component of Xu and Chun's identification process, as the CDA at this site is not enhanced on Sep trials relative to Solo where three features also require identification. Future work combining EEG and fMRI methods and/or magnetoencephalography (MEG) is needed to determine whether CDA activity at distinct electrode sites is associated with iIPS and sIPS/LOC activation within individuals, and across different feature contexts and task demands (i.e., change-detection tasks using delays and motion as visual disruption). 

Representations in VSTM during maintenance

Given the latency of the CDA (onsets at 400 ms) and that this component is believed to reflect the maintenance stage of VSTM, this study cannot directly address questions about how or in what format perceptual information is initially encoded into VSTM. However, the present results suggest that feature load effects on the CDA differ depending on feature location—i.e., features presented at one location versus separated by distinct locations—and the electrode observed. In other words, the CDA appears to be sensitive to the binding of features within object files as well as individuating discrete object files. Fruitful follow-up work might take advantage of these dissociable effects in the CDA, addressing questions in the object file literature, such as those posed by Treisman and Zhang (2006) and Hollingworth and Rasmussen (2010)—namely that location plays an important role to VSTM, serving to initially bind features into object files; however, over the course of consolidation, features may gradually become location independent, enabling more dynamic representations and object tracking. More specifically, this work may focus on changes in the CDA at sites P1/P2 and P7/P8 at earlier and later time windows, and how this may interact with task demands—i.e., under conditions in which it may be advantageous to bind versus dissociate features and across different visual perturbations classically used in both VSTM and object file paradigms, such as delays and motion. Also, whether there are two distinct neural systems that underlie these electrophysiological results and the degree to which these systems may be codependent remains an open question for further exploration. 

Conclusion

The results presented here highlight the utility of the CDA in exploring VSTM representations. The dissociable effect of presenting multiple features bound at one location compared to separate locations provides electrophysiological support for a neural object file theory and a dual-system model of VSTM. We interpreted the behavior of the CDA at electrode pair P1/P2 as reflecting the process of object individuation and at P7/P8 as reflecting object identification or feature binding. Toward a richer understanding of the nature of representations in VSTM, our results highlight the need to go beyond analyzing the topographical peak of activation or averaging across all posterior electrodes that show a CDA, as these methods may fail to capture the more fine-grained distinctions afforded by this event-related potential. It is critical to consider not only the number of objects/stimuli, but also within-object features when investigating VSTM capacity as both appear to be distinctly represented, eliciting dissociable effects on the CDA at separate topographical regions. These dissociable load effects on the CDA at different electrode sites may specify divergent capacity constraints with one system determining the upper-limit of discrete objects that can be stored and individuated in VSTM, whereas the other system places further constraints on the resolution or number of within-object features made available for identification. 

Acknowledgments

The authors would like to thank Megumi Noda and Vince Brienza for their assistance in data collection of the full probe version of the task. This research was supported by grants awarded to Susanne Ferber from the Canadian Institutes of Health Research (MOP106436), the Natural Science and Engineering Research Council of Canada (261203), and the Early Researchers Awards Program of the Province of Ontario. 

Commercial relationships: none. 

Corresponding author: Kristin E. Wilson. 

Email: [e-mail]. 

Address: Department of Psychology, University of Toronto, Ontario, Canada. 

Footnotes

¹  Degrees of freedom and significance adjusted using Huynh-Feldt correction for violation of sphericity.