Relational information in visual short-term memory: The structural gist

Abstract

Over the past 20 years, storage of visual items in visual short-term memory has been extensively studied by many research groups. In addition to questions concerning the format of object storage is a more global question that focuses on the organization of information in visual short-term memory. In a series of experiments we investigated how relations across visual items determined the accessibility of individual item information. This relational information seems to be very strong within the store devoted to each feature dimension. We also investigated the role of selective attention on the storage of relational information. The experiments suggest a broadening of the parallel store model of visual short-term memory proposed by M. E. Wheeler and A. M. Treisman (2002) to include the notion of what we call “structural gist.”

Introduction

Understanding the nature of the information stored in visual short-term memory (VSTM) leads to two questions. First, what is the format of the storage units? Second, how are the units organized in visual short-term memory? 

The scientific community is still discussing the first issue concerning the storage units. Certain authors defend an object-based unit (Luck & Vogel, 1997; Vogel, Wood-man, & Luck, 2001; for a review, see Scholl, 2002), while others favor a feature-based one (Wheeler & Treisman, 2002) where features characterizing an object are not bound together in memory. The two views oppose each other in the way they explain the capacity of visual short-term memory. In the object-based approach, the storage capacity is determined by the number of objects, each of which can contain a large number of bound features. On the other hand, storage capacity in the feature-based theory is limited by the maximum number of features of a given dimension that can be stored simultaneously in parallel feature-specific memory stores. 

Many studies favor the feature-based theory in that they observe independence across the different features of an object. This evidence derives from tasks such as similarity judgment of stimuli (Handel & Imai, 1972), sorting stimuli (Gottwald & Garner, 1972), recognition memory (Stefurak & Boynton, 1986), and partial-report (Isenberg, Nissen, & Marchak, 1990). Until now, the idea that features are not bound in visual short-term memory is a dominant view in the literature, and evidence for the object-based view has not been replicated (Delvenne & Bruyer, 2004; Olson & Jiang, 2002; Wheeler & Treisman, 2002; Xu, 2002). Nevertheless, debate about the identity of the memory unit is still open. 

The second question, concerning the organization of units in visual short-term memory, is the focus of this study. Jiang, Olson, and Chun (2000) have shown that detection of changes in featural information depends on the invariance of the spatial configuration of the displayed items, thereby suggesting that units coded in a given presentation are not stored independently but rather as a function of the whole stimulus configuration. Indeed, using a variant of the change detection paradigm, Jiang et al. (2000) observed that when non-targets disappear during a blank interval between two successive stimulus frames (Figure 1, top), this decreases the ability of subjects to detect a change in the color of a cued target object. Moreover, when the spatial configuration of the presented objects changes during the interval (Figure 1, bottom), this interferes with fea-ture change detection, whereas when features change this does not impair spatial change detection. This asymmetric relationship across spatial and feature information of objects supports the idea of a configuration-based relation between single features in visual short-term memory. 

Figure 1

 

Samples of displays and experimental sequences used in Jiang et al. (2000). Figures are not drawn to scale. Top: sup-pressed configuration; middle: same configuration; and bottom: different configuration.

Our purpose is first to characterize the nature of such relational information and second to evaluate the role of attention in its establishment. We shall first consider the question concerning the nature of the relational information. 

If, as Jiang et al. (2000) suggest, spatial configuration is the framework supporting visual short-term memory, what kinds of information are linked together? Because color change detection performance decreases when non-target and target vary on the same color dimension (Jiang et al., 2000, preliminary experiment not reported in detail by authors) (Figure 2, top), we can conclude that color information characterizing different items interacts. 

Figure 2

 

Samples of displays used in Experiments 1–8. From top to bottom, Experiments 1, 4, 5, 7, and 8. Figures are not drawn toscale. Experiment 1 is the same as in the preliminary experiment in Jiang et al. (2000).

Is this true for other feature dimensions? And can such an interaction occur between different feature dimensions? 

To answer these questions, we conducted five experiments using the same paradigm as Jiang et al. (2000). This paradigm consisted of designating with a cue box a target among all the presented items when the test screen appeared. Two kinds of changes could be made: a minimal change (only target features change across the sample screen and the test screen) and a maximal change (where all non-targets could change features). Experiments 1–3 were con-ducted to reproduce the Jiang et al. preliminary color find-ings and extend them to other dimensions, such as orientation and shape. In these experiments target and non-target feature changes were restricted to a single feature dimension, whereas in Experiments 5 and 6 the changing dimensions of target and non-targets were different. 

Because in experimental protocols of this kind the target and the non-targets change simultaneously, the observed effects can be interpreted as being the result of an increase of noise in the baseline (in minimal change condition, observers have to differentiate 1 change from 0 changes vs. N changes from N — 1 changes in the maximal change condition; N is the number of items). To face this theoretical interpretation of the data, we conducted Experiment 4 that tries to distinguish an explanation in terms of relational information from an explanation in terms of noise increment in the signal baseline. 

In the paradigm used in these experiments, all items presented in the first screen can potentially become the cued target item on the second screen; so they all need to be attended in the dimension of change. How do we store relational information if we select only a part of the presented information? When some items have to be separated attentionally from a group of distractors before encoding, do they still suffer from interference in change detection when the distractors change their feature value? The second main question we asked in this study was whether the relational information in visual short-term memory exists only across objects that have been attended in the dimension of change, or if it also links those items to ignored ones. In this latter case, we could conclude that relational information follows processing rules other than those applying to individual information. Experiments 7 and 8 were completed to extend the results of Experiments 1 to 4 concerning the role of attentional processes for encoding the relational information. 

Experiments 1, 2, and 3

To investigate and generalize preliminary findings on color by Jiang et al. (2000), we extended the minimal change/maximal change paradigm to the shape and orientation dimensions. 

Methods

Participants

In all experiments reported here participants were university students who volunteered. All had normal or corrected-to-normal visual acuity and normal color vision. For each experiment we used a different group of subjects. In this first experiment, 10 observers were used for the color variant, 10 for the shape variant, and 8 for the orientation variant. At the beginning of the experiments participants were given a detailed description of the study. Twenty practice trials before the experiment allowed participants to familiarize themselves with the experimental design. At the conclusion of the study each participant could ask the experimenter questions and give his or her impressions. 

Experimental procedure

On each trial, subjects viewed a sample array and a test array separated by a brief delay. On every test array one item was marked with a cue box (target item), and the subjects had to decide whether the item had changed on a dimension previously specified to the subject (Figure 2) compared with the sample array. In half the trials the other items in the test array (non-target items) changed in the same feature dimension. Following Jiang et al. (2000) we called this the maximal change condition. When the non-target items did not change, we called this the minimal change condition. The experiment lasted approximately 40 min to 1 hr and was divided into blocks of trials where all the conditions were randomly mixed. Between blocks subjects could pause for a few minutes if they wanted. For the color condition we used 4 blocks of 100 trials (400 = 2 non-target change conditions [maximal vs. minimal] × 2 target change conditions [change vs. no change] × 4 set sizes [2, 4, 6, 8] × 25 trials). For orientation and shape conditions, we used 4 blocks of 120 trials because we reduced the number of set sizes to 2, 4, and 6 items and had 40 trials per condition. Location of all items in the arrays was constant during each trial. The only change that could occur from the sample to the test array happened in a single feature dimension (color, shape, or orientation). The sample screen was presented for 100 ms and after a delay of 1000 ms with a white background, the test screen was presented for 2000 ms. When the test screen appeared the subjects had the possibility to answer whenever they wanted. 

After pushing a button on a keyboard to indicate their response, subjects had to again push a button to initiate the next trial. Each trial started with a central black fixation dot lasting 1000 ms. To avoid the possibility of verbal coding of items, we integrated a verbal load task that consisted in repeating aloud a randomly chosen pair of vowels, which appeared for 500 ms between the fixation dot and the sample screen. These vowels had to be verbally repeated at the end of each trial. Errors were very rare (<1%). When the vowels disappeared, a 500-ms grey screen was presented before onset of the sample screen. 

Stimuli and apparatus

All experiments were programmed and executed using MATLAB 6.5.0 with the Psychophysics Toolbox extensions (Brainard, 1997; Pelli, 1997). Each item was randomly positioned in an invisible 4 × 3 cell matrix region (9.8° × 7.3°) on a video monitor with a grey background. The position of each item in a cell was slightly jittered. For the color experiment we had seven different colored squares (red, blue, green, purple, white, black, and yellow). With set size 8 there was one color that repeated among the presented items. For the orientation experiment we used four differ-ent orientations for the black bars (size 0.03° × 1.15°): 0°, 45°, 90°, and 135°. For the shape condition we used nine different black shapes (square, circle, triangle, hourglass, vertical rectangle, horizontal rectangle, cross, U-up, and U-down). The cue consisted of a light-brown square 0.97° × 0.97° with a line thickness of 0.03°. Participants were seated at a distance of 70 cm from the screen. They were tested individually in a room with normal lightening. 

Results

Correct answers for target change detection were assessed as a function of non-target change conditions (maximal vs. minimal change) and set sizes (2, 4, and 6 items, and 8 for color condition) (Figure 3). We also calculated the mean sensitivity and response bias (ß value) for each condition. 

In all three experiments we obtained a significant difference in performance for the set size factor and for the non-target change conditions factor (Figure 3). Performance is significantly poorer for the maximal change condition. Mean values (across the different set sizes) for differences across minimal and maximal change conditions are 12% for the color condition, 7% for the orientation condition (Figure 3), and 5% for the shape condition (Figure 3). We have the following ANOVA results for non-target change conditions: F(1,9) = 42.6, p < .01 for the color condition, F(1,7) = 131.8, p < .01 for the orientation condition, and F(1,9)=24.7, p < .01 for the shape condition. ANOVA re-sults for set size effects are F(2,18) = 131.8, p < .01 for the color condition, F(2,14) = 98.2, p < .01 for the orientation condition and F(2,18) = 166.1, p < .01 for the shape condition. These results reproduce the set size dependent capacity limit effect of visual short-term memory observed in the literature (Luck & Vogel, 1997; Pashler, 1988; Phillips, 1974; Vogel et al., 2001). ANOVA results for the interaction across set size and non-target change factors for color, orientation and shape are, respectively, F(3,27) = 6.6, p < .05, F(2,14) = 11.1, p < .05, and F(2.18) = 3.6, p < .05. Post hoc analyses were performed by means of the New-man-Keuls test and revealed that for color and orientation conditions the non-target change factor has a significant effect only for set sizes 4 (p < .01) and 6 (p < .01), whereas for the shape condition the significant effect is observed for set sizes 2 (p < .01) and 4 (p < .01). 

Figure 3

 

a. Experiment 1 (color). b. Experiment 2 (orientation). c. Experiment 3 (shape). Percentage of correct responses for target change detection as a function of non-target change conditions (maximal change-interrupted line vs. minimal change-continuous line) and set sizes. d. Experiment 4. Percentage of correct responses for target change detection as a function of blank item change conditions (maximal change-interrupted line vs. minimal change-continuous line) and set sizes. The bars represent the confidence interval associated with each condition.

To be certain that the above results were not the expression of different response criteria in the different non-target change and set size conditions, we calculated sensitivity (d′ value) and response criterion (β value). Sensitivity values confirm performance observed in percentage correct: There are significant differences across minimal and maxi-mal change conditions [color condition: F(1,9) = 16.5, p < .05; orientation: F(1,7) = 13.08, p < .05; and shape condition: F(1,9) = 10.9, p < .05], and performance was significantly worse with increasing set size values [color condition: F(2,18) = 48.6, p < .01; orientation: F(2,14) = 85.9, p < .01; and shape: F(2,18) = 94.1, p < .01]. β values were significantly more liberal in the maximal change condition for the color condition, F(1,9)=9.1, p < .05, but for neither of the other conditions [orientation: F(1,7) = 2.8, p > .05; shape: F(1,9) = 2.2, p > .05] nor for the different set size conditions [color: F(2,18) = 1.7, p > .05; orientation: F(2,14)=2.2, p > .05; shape: F(2,18) = 2.1, p > .05]. 

Discussion

In all three experiments, we found a significant decrement of change detection performance in the maximal change condition, with the effect appearing earlier for the shape condition but being more pronounced in the color condition. These results are corroborated first by the significant loss of sensitivity (d′ value) observed for the maxi-mal change condition and second by the absence of a difference in β values in Experiments 2 (orientation) and 3 (shape), which implies that response strategies are not involved. In Experiment 1 (color), we found a significant ß-value difference across minimal and maximal change conditions: This more liberal tendency for the maximal change condition could explain the stronger effect observed for the color condition. However, this β-value difference probably does not reduce the maximal change effect to a strategy effect because the maximal change effect was observed for orientation and shape conditions independently of value variations. 

Finally, the difficulty in change detection induced by the modification of information surrounding the target item implies that this contextual information is necessary for effective retrieval of target information. We obtained the same effect as Jiang et al. (2000) obtained in their non-target spatial configuration change condition. So it seems that to be correctly retrieved, the presence of the correct contextual information about individual items in the correct configuration is necessary. 

These results raise two questions. First, target and non-target changes happened in the same dimension. Would we observe the same kind of dependence if the changes occurred in different dimensions? We studied this point in Experiments 5 and 6. The second and critical point concerns whether the results should be interpreted in terms of contextual dependencies or in terms of a noise effect. In-deed, because the changes of the non-targets are simultaneous with the target detection, the observed differences across conditions can be attributed to an increment in the baseline noise as expressed in psychophysical terms. Following Weber’s Law on signal strength and stimulus detection, it might become harder to detect a change in an environment where many changes occur at the same time (Green & Swets, 1966). Jiang et al. (2000) had already proposed this interpretation of their maximal/minimal change experiment on color without resolving the problem. When-ever a feature of non-target changes simultaneously with target change, detection deficits can be interpreted as caused by an increase of noise in the baseline instead of an impairment in relational information. This last point is crucial if we want to show the existence of this relational information. In the next experiment we propose a solution to distinguish between the noise model and a model postulating the existence of non-spatial relational information. 

Experiment 4

As noted above, what defines noise is presumably the simultaneous change of the non-targets with the target change. If a non-target changes well before the moment of target detection, with the subjects knowing which item (be-sides the target) will be different at test screen, will it affect performance in the same way as it did in Experiments 1–3? A noise model would predict no decrease in performance. If there are no changes occurring simultaneously to non-targets and the target, the change detection on the target should not be altered. This advanced change of a non-target can be perceived and integrated by the subjects and does not represent new information when the test screen appears. If there is relational information in VSTM and it contributes structurally to the target information, then this non-target change will affect the performance in its change detection. 

Methods

Participants

Twelve university students who volunteered participated in the experiment. As in the previous experiments they were given practice trials before the test session began. 

Experimental procedure

To assess the prediction made above, we used the same procedure as used for Experiments 1–3 but with a modification. When all items (colored squares) disappeared at blank screen, one item remained on the screen: For easy reference we will call it the blank item (Figure 2). This item could change in the transition from sample screen to blank screen. This factor determines the two conditions. When the test screen appeared the only item that could change was the target item. All other non-targets, including the blank item, remained the same. To avoid the possibility of subjects excluding the blank item from their global scene integration at test screen, this item could also be a potential target. Subjects thus had no alternative but to attend all items presented in each trial. The responses corresponding to the blank item as target are excluded from the final analysis because they correspond to the detections on items that are always in view, and thus do not depend on the information stored in memory to be recalled. As in the previous experiments, the target was marked with a cue box at test screen. The experiment lasted for about half an hour. It consisted of one block of 200 trials where all the conditions were randomly mixed (200 trials = 2 blank item change conditions [change vs. no change] × 2 target change conditions [change vs. no change] × 2 set sizes [4 and 6] × 25 trials). 

Stimuli

The items were colored squares. Background and cue were the same as in Experiment 1. We replaced white with pink, because during the blank screen this color would have merged with the background and an eventual non-target would have disappeared. Cyan was added to constitute eight possible colors to avoid repetitions. The stimulus arrays were composed of 4 or 6 items of different colors. 

Results

One subject was excluded from the analysis because the response pattern was completely inverted with the response pattern of all other participants, and was thus considered to have inverted the response buttons at some point in the experimental block. Figure 3 shows the percentage of correct answers for target color change detection as a function of blank item change condition (change vs. no change) and of set size (4 and 6). We applied an ANOVA and observed a significant main effect of the blank item change, F(1,11) = 20.1, p < .01. The analysis of sensitivity values d′ showed the same pattern of results: The blank item change factor had a significant effect on sensitivity, F(1,11) = 6.64, p < .05. The β-value analysis shows no change of response strategy with blank item change, F(1,11) = 1.11, p > .05. 

As concerns the set size factor, a significant main effect is found, F(1,11) = 65.7, p < .01. Sensitivity values d′ were significantly worse with increasing set size, F(1,11) = 32.65, p < .01. β values show no change in increasing set size, F(1,11) = .95, p > .05. We found no significant effect of interaction between distractors change and set size factors, F(1,11) = .14, p > .05. 

Discussion

There is a clear decrease in performance in the condition where the blank item changes color. This decrease in performance is interpreted as a consequence of a change in relational information. Nevertheless, we should consider two alternative interpretations. First, it could be argued that the changed blank item is considered as new information within the test screen layout. This would mean that even though the feature value of the item is known in advance, it is considered as new in a new context. This could eventually be interpreted in terms of the noise hypothesis, but at the same time it implies that the change detection depends on the feature values of other neighboring items in the scene. This possibility would not contradict our pre-diction but rather support it. A second alternative explanation we considered was once the blank item has changed, its interaction with the stored items creates memory impairment in VSTM during the delay. This would require a relation between a directly perceived item and items in VSTM. For this to be possible, all items would have to interact. Again, this is saying that relational information is important in target evaluation. The overall performance decrease at set sizes 4 and 6 compared to Experiment 1 can be explained by the general disturbance created by the unpredictable presence of one of the items in the sample screen at blank screen. It does not explain the difference found between the conditions. In summary, for the reasons mentioned, we believe that the results of this experiment argue for the existence of feature relational information in VSTM. 

Experiments 5 and 6

The following question now arises: In Experiments 1–3 we made a change in a unique feature dimension but would we observe the same kind of dependence if target and non-target changes occurred in different dimensions? 

If, following Jiang et al. (2000), spatial configuration is the framework supporting visual short-term memory, what kinds of information are linked together? The previous experiments showed that information characterizing items along a single dimension is linked together in visual short-term memory. But is information from different dimensions also linked together in VSTM? 

Many studies support the idea that features belonging to the same item (e.g., shape and color) are not bound in visual short-term memory (Gottwald & Garner, 1972; Handel & Imai, 1972; Isenberg et al., 1990; Stefurak & Boynton, 1986, among others). Others show also that the requirement to bind features can reduce the capacity of visual short-term memory (Wheeler & Treisman, 2002), implying that item information is not stored in a bound format by default. 

If at the item level the memory for a given dimension (e.g., color) can be independent of the information concerning another dimension (e.g., shape), a fortiori we would expect little dependence when the different dimensions belong to different items. Consistent with this, Rensink (2000), using a flicker paradigm in a study on attention and change detection, notes that polarity change of non-target items does not impair change detection performance for orientation of the target. These results lead us to predict that we will not observe any decrement in performance when a non-target changes in a dimension other than in the dimension of change of the target. To test this assumption, we used the same experimental design as used for the three previous experiments but with crossed dimensions. 

Methods

Participants

Twenty-four university students who volunteered participated in Experiment 5, and 10 students who volunteered participated in Experiment 6. As in the previous experiments, they were given 20 practice trials before the test session began. 

Experimental procedure

To assess the cross-dimensional hypothesis, we used the same procedure as was used for Experiments 1–3. Here we test whether change detection of a cued item in a given dimension (Experiment 5: color; Experiment 6: shape) is impaired by the change of non-target items in another dimension (Experiment 5: shape; Experiment 6: color). Experiment 5 is composed of two parts: a crossed dimension part and a single dimension part. In the crossed dimension part, the target item can change color and non-targets can change shape, whereas in the single dimension part both target and non-targets can change color. The sin-gle dimension part was nearly the same as Experiment 1: The only difference was that all items had different shapes, whereas in Experiment 1 they were all squares. The experiment lasted for about 1 hr. Each experimental part was divided into two blocks of 120 trials where all the conditions were randomly mixed (240 trials = 2 non-target change conditions [maximal vs. minimal] × 2 target change conditions [change vs. no change] × 3 set sizes [2, 4, and 6] × 20 trials). Half of the participants started the experiment with the crossed dimension part and the other half with the single dimension part. 

We conducted the sixth experiment to verify if the pat-tern of results observed in Experiment 5 still exists when the dimensions are inverted (target can change shape; non-targets can change shape); thus, we reproduced only the crossed dimension part of Experiment 5. 

The experiment lasted for about half an hour and was composed of 240 trials (2 non-target change conditions [maximal vs. minimal] × 2 target change conditions [change vs. no change] × 3 set sizes [2, 4, and 6] × 20 trials). 

Stimuli

The shapes and colors of objects, background, and cue were the same as in Experiments 1 (color) and 3 (shape). Here the stimulus arrays were composed of 2, 4, or 6 ob-jects of different shapes and colors (Figure 2). There was no color repetition in the presented items. 

Results

From Experiment 5, Figure 4 shows the crossed dimension and the single dimension parts and the percentage correct answers for target color change detection as a function of non-target change condition (maximal vs. mini-mal change) and set size. A first ANOVA revealed a significant interaction, F(1,23) = 6.8, p < .05, between the non-target change factor (maximal vs. minimal change) and the dimension factor (crossed vs. single): The non-target changes did not have the same effect when they involved the same dimension as when they involved different dimensions. 

Figure 4

 

a. Experiment 5. Percentage correct response for target change detection as a function of non-target color change condition and set size, single dimension part; crossed dimensions part (b). c. Experiment 6. Target shape change detection as a function of non-target color change and set sizes. d. Experiment 7. Percentage correct responses for target color change detection, as a function of distract or change conditions and set sizes. e. Experiment 8. All maximal changes are represented by interrupted lines, and all minimal changes are in continuous lines. The bars represent the confidence interval associated with each condition.

We applied an additional ANOVA to each experimental part and observed a significant main effect of the non-target change but only in the single dimension part, F(1,23) = 23.4, p < .01, and not in the crossed dimension part, F(1,23) = .48, p > 05. The analysis of d′ values showed the same pattern of results: The non-target change factor had a significant effect on sensitivity in the single dimension part, F(1,23) = 11.9, p < .01, but not in the crossed dimension part, F(1,23) = .42, p > .05. But for the β-value analysis we observed that the non-target change had a significant effect on β value in the single dimension, F(1,23) = 7.3, p < .05, and also in the crossed dimension part, F(1,23) = 6.7, p < .05. So these results show that subjects adopt a more liberal response criterion for the maxi-mal change condition in the single than in the crossed dimensions parts. However, this tendency is present in the two experimental parts with the same strength: The strategy effect cannot be responsible for the non-target change effect because in the crossed dimensions part it was not sufficient to cause a significant decrease in performance in the maxi-mal change condition. 

As concerns the set size factor, a significant main effect is found in the crossed dimension, F(2,46) = 108.5, p < .01, and in the single dimension part, F(2,46) = 137.8, p < .01. d′ values were significantly worse with increasing set size in the single dimension, F(2,46) = 123.8, p < .01, and in the crossed dimensions part, F(2,46) = 81.8, p < .01. β values were significantly more conservative with increasing set size in the single dimension, F(2,46) = 10.5, p < .01, but not in the crossed dimensions, F(2,46) = 1.7, p >.05. 

In the single dimension part we found no significant effect of interaction between non-target change and set size factors, F(2,46) = .54, p > .05. To understand the absence of interaction we conducted post hoc analyses (Newman-Keuls): The effect of the non-target change factor appears to be significant for each set size (p < .05 for set size 2 and p < .01 for set sizes 4 and 6). 

Figure 4 shows in Experiment 6 the percentage of correct responses for target shape change detection as a function of non-target color change condition (maximal vs. minimal change) and of set size. We applied an ANOVA and observed no significant main effect of the non-target change, F(1,9) = .11, p > .05. The analysis of d′ values showed the same pattern of results: The non-target change factor had no significant effect on sensitivity, F(1,9) = .92, p > .05. The β-value analysis shows no change of response strategy with non-targets change, F(1,9) = 2.16, p > .05. As concerns the set size factor, a significant main effect is found, F(2,18) = 161.62, p < .01. d′ values were significantly worse with increasing set size, F(2,18)=88.39, p < .01. β val-ues show no change with increasing set size, F(2,18) = 1.7, p > .05. We found no significant effect of interaction between non-target change and set size factors, F(2,18) = 1.45, p > .05. 

Discussion

As concerns the main question posed, the data from the crossed dimension conditions show that observers’ detection performance for target change on one dimension was not affected when the non-target items changed on the other dimension. Moreover, the single dimension condition of the Experiment 5 confirms the results of Experiment 1 (color version) but with an array of differently shaped items. 

We can conclude that when one dimension is at-tended, changes happening in another dimension do not impair change detection. Because attention seems to play an important role in determining the creation of relational information, we will study this question in Experiments 7 and 8. 

Experiment 7

In the preceding experiments the establishment of relational information between items in visual short-term memory could be a consequence of their mere presence in the scene. According to Experiments 1–3, this possibility is plausible but only within a given dimension: The presence of a feature in the crossed dimension parts of Experiments 5 and 6 was not sufficient to influence the retention of in-formation belonging to another dimension. But because we observed that dependency effects appear only when target and non-target changes happen in an attended dimension, an alternative could be that the dependency appears only between sources of information that are relevant for the task. In experiments until this point, all items in the display had to be attended. We wanted to investigate a new theoretical hypothesis: Does spatial selective attention deter-mine the relational links between the items in visual short-term memory? 

It is known from studies in visual search (Duncan & Humphreys, 1989) that the time taken to find a target among a group of distractors is inversely related to the similarity between targets and distractors. The selective attention process will be more or less effective depending on whether it can rely immediately on a parallel search of the scene (and find the target by a pop-out effect) or if it be-comes engaged in a serial search of each item present in the scene. 

Jiang et al. (2000) did another experiment to see if a change of location in a group of distractors could affect the color change detection on a group of targets. Their results showed that there was no effect of distractor location changes on target color change detection. The distractors were all the same color (white) and contrasted highly with the targets that were colored. This strong difference created a pop-out effect with very little top-down processing that needed to be brought into play to separate potential targets from distractors. If both types of items would require more effort to be distinguished from one another in the selection, would there be any influence of the changing distractors on the target’s detection? 

We wanted to investigate this matter further, especially as concerns the role of top-down attention in the establishment of relational information. In particular, is relational information limited to attentionally attended elements in space? Avoiding pop-out effects, we asked the following question: When only some elements in the display need initially to be attended in the encoding phase, does changing the unattended items still interfere with performance? 

Methods

Participants

Ten student volunteers participated in this experiment. Twenty practice trials allowed subjects to familiarize them-selves with display and task demands. 

Experimental procedure

To test the role of top-down attention in creating relational links between features, we used a variant of the minimal/maximal change paradigm. Subjects had to make a visual segmentation to distinguish potential targets from distractors. To make this distinction, the subject was in-formed of the shape of the target category at the beginning of each trial. By means of a grey target shape, which appeared next to the two white vowels involved in the articulatory suppression task, subjects were informed at the be-ginning of each trial of the shape of the target they had to distinguish. This screen appeared for 1 s instead of its usual 500 ms. In the sample array subjects had to attend to the elements belonging to the set of potential targets that had been indicated by the grey indicator. They had to do this search quickly before the blank screen appeared. In the test array, they had to detect a change among potential targets (Figure 2). On half the trials all distractors changed colors. The potential targets, which finally are not the target, never change color. 

The experiment lasted about 45 min. We used 360 trials (360 = 2 distractor change conditions [maximal vs. minimal] × 2 target change conditions [change vs. no change] × 3 potential targets set sizes [2, 4, and 6] × 30 trials) divided into 3 blocks of 120 trials where all the conditions were randomly mixed. 

Stimuli

The shape-cue for the target changed randomly at every trial so subjects would not be facilitated by one type of shape. The shapes we used were square, triangle, circle, and cross. The dimensions of the shapes were the same as in Experiments 3, 5, and 6. The colors were the same seven colors as in Experiments 1, 5, and 6. The presentation times of the sample and test arrays were the same as in the previous experiments. 

For each set size (2, 4, and 6) of potential targets there was the same number of distractors. They were randomly placed in the layout. Contrary to the experiment of Jiang et al. (2000), to make the search effortful, the potential targets and distractors shared the same possible colors. Thus the distinction had to be made only on the basis of shape, eliminating a possible pop-out effect. 

Among the potential targets no single item was marked by a cue box, contrary to previous experiments. The task was to keep in memory the colors of the potential targets to see if one of them had changed between the sample array and the test array. The use of a cue box was unnecessary here because in past experiments of single item change detection among an unchanging group of visual items such cueing was shown to have no effect (Vogel et al., 2001). 

Results

Figure 4 shows the percentage correct for target color change detection, as a function of distractor color change condition (maximal vs. minimal change) and as a function of set size. As in Experiments 1–4, we observed a significant difference in performance for distractor changes, F(1,9) = 20.1, p < .05, and for set size factors, F(2,18) = 115.4, p < .01. Analysis conducted on d′ values confirm observed performance: There were significant differences in d′ values between minimal and maximal change conditions, F(1,9) = 13.7, p < .05, and they are significantly worse with increasing set size, F(2,18) = 106.7, p < .01. Distractor changes had no effect on β values, F(1,9) = 1.2, p > .05, but increasing set size leads significantly to a more conservative criterion, F(2,18) = 4.03, p < .05. 

There was a significant interaction between the two fac-tors, F(2,18) = 9.17, p < .05. Post hoc comparisons show that, as in Experiments 3 and 5, the effect of the distractor change factor appears from set size 2 (p < .01). Contrary to Experiment 1 and to the single dimension part of Experiment 5, post hoc comparisons show that there is no significant effect of distractor change for set size 6 (p > .05). To ensure that the visual search task was effortful, we gave no more time to subjects than in the previous experiments. We were conscious that it would be easier to make the distinction between targets and distractors at smaller set sizes than at larger ones. We interpret the percentage correct results for both conditions at set size 6 as being due to this limiting time factor. 

Discussion

We observed in Experiment 7 that changes in information conveyed by items irrelevant for the task impair change detection performance. We can thus explain the existence of relational information found between items in Experiments 1–4 as deriving from their mere presence in the scene. But because we noticed in Experiments 5 and 6 that the changes of information belonging to a given dimension had no effect on detection of change in another dimension, we cannot generalize the conclusion to all kinds of information present in the scene. We thus conclude that attention does not determine the existence of relational links between information belonging to the same dimen-sion (Experiments 1–4) in the way for information belong-ing to different dimensions (Experiments 5 and 6). 

However, these results can be criticized on a major point: The way we designed the experiment avoided the strong pop-out effect studied by Jiang et al. (2000) but does not allow us to assert that the subjects performed the task without attending to the irrelevant information. Indeed, the search subjects had to carry out required that they had to check each item to determine if it was relevant or not for the task. Even if the attentional allocation necessary for this evaluation is minimal, we cannot exclude that it might be sufficient to create relational information between relevant and irrelevant items. Moreover, we had at our disposal seven different colors for a maximum of 12 visual items on the screen: There was some overlap between color information for relevant and irrelevant items. This could have induced a grouping effect that might have directed attention to little groups composed of relevant and irrelevant items. We conducted Experiment 8 to avoid these visual search and grouping effects. 

Experiment 8

To be sure only the relevant items were attended, we needed to automatically direct the attention toward the potential targets. Studies about the control of visual attention showed that the abrupt onset of a visual stimulus automatically attracts attention to its location (Posner, 1978). The attentional shift initiated is described as “exogenous” (Posner, 1978) or “involuntary” (Jonides, 1981; Luria, 1973; Müller & Rabbitt, 1989) and is difficult to suppress (e.g., Jonides, 1981, 1983; Müller & Rabbitt, 1989). By using this irrepressible attentional shift effect we could direct and restrict the subject’s attention to the potential targets. Moreover, Wright (1994) studied multiple simultaneous location cues and observed that when items are presented in between cued locations their identification response times are not reduced. These results imply that cued locations are not processed by an attentional focus of variable spatial extent that encompasses multiple cued locations and allow us to be sure that the precue benefit exists specifically for the precued items. Thus, in Experiment 8 we added direct cues to the top-down attentional cue previously provided in Experiment 7. Furthermore, we avoided the presence of repeated colors within a scene to prevent pop-out effects that could have led to grouping of relevant with irrelevant items. 

Methods

Participants

Ten student volunteers participated in this experiment. Twenty practice trials allowed subjects to familiarize them-selves with the display and the task demands. 

Experimental procedure

We used the same paradigm as in Experiment 7. The subject was again informed at the beginning of each trial of the shape of the target category with the onset of the grey target shape next to the two white vowels. At 100 ms before the apparition of the sample screen we inserted a precue screen for 60 ms. The target changed color on half the trials. This variable was crossed with the variable of change of the distractors, also on half the trials. The other potential targets never changed color on the test screen. The experiment lasted about 45 min. We used 360 trials (360 = 2 distractors change conditions [maximal vs. minima] × 2 target change conditions [change vs. no change] × 3 set sizes of potential targets [2, 4, and 6] × 30 trials) divided into 2 blocks of 180 trials where all the conditions were randomly mixed. 

Stimuli

To avoid repetition we increased the number of colors available to 14 (red, blue, green, purple, white, black, yellow, + orange, light blue, pink, brown , light pink, dark green, and grey). We used the same shapes as in Experiment 7. The direct cues were white dots positioned at the middle of the space occupied by the precued item (Figure 2). The target cue again consisted of a light-brown square. 

Results

We applied an ANOVA and observed a significant main effect of the distractors change, F(1,9) = 76,97, p < .01. The analysis of sensitivity values d′ showed the same pattern of results: The distractors change factor had a significant effect on sensitivity, F(1,9) = 17.8, p < .01. The β-value analysis show no change of response strategy with distractors change, F(1,9) = .61, p > .05 (Figure 4). As con-cerns the set size factor, a significant main effect is found, F(2,18) = 196,78, p < .01. d′ values were significantly worse with increasing set size, F(2,18) = 168.75, p < .01. β values show no change with increasing set size, F(2,18) = 2.29, p > .05. We found a significant effect of interaction be-tween distractors change and set size factors, F(2,18) = 8.51, p < .01. We observed the same global decrease in perform-ance as in Experiment 7. 

General discussion

Our results are in agreement with past results (Jiang et al., 2000) and go further in the comprehension of the organization of information in visual short-term memory. Although Jiang et al. introduced the notion of relational information in visual short-term memory concerning spatial information, the results shown here point to an extended and complementary idea of relational information that concerns item features. This relational information of features seems to always link information within a given feature dimension (Experiments 1–3) irrespective of spatial selective attention (Experiments 7 and 8) but not by default across dimensions (here tested between color and shape in Experiments 5 and 6). These results suggest a new aspect that must be added to the classical view of visual short-term memory in which memory is considered mostly as a repository of individual elements, items, or features. 

As mentioned earlier, experiments using co-occurring changes of target and non-targets generate results that might be explained by a noise model. The results of Experiment 4 show that this observed interference probably cannot be explained with the classical definition of noise leaving; thus, an explanation in terms of relational information is possible. 

Based on the results obtained in this present study, we will now try to give a more precise description of relational information. When simultaneously storing a variety of items in VSTM, two types of information will be encoded. A first kind of information concerns the whole scene and consists of a uniquely determined combination of all the individual feature characteristics present at the moment of encoding (e.g., all colors or all shapes). It is not restricted by spatial selective attention and extends in each dimension to all items present in the scene. This information about the whole scene is an integral part of local information and is what we call relational information. We thus assume that any information from a given spatial locus is the synthesis of individual information (such as “red” or “round”) and relational information (red-and-green-and-blue or circle-and-square-and-triangle). In our experiments we observed that changing individual information about non-targets affects the change detection of an indicated target (Experiments 1–4). This change in a feature of the non-targets results in a change in the relational information that is part of the local target information. Because of this “internal” change in all local information of the display, target detection is impaired. Because the relational information is an aspect of the whole scene in that it contains a synthesis of all individual features, and thus represents relationship information, observers cannot describe it in an explicit form. When we keep a variety of items in VSTM, the set of all the relational information within all the individual items information forms what we call the “structural gist.” 

The notion of structural gist that serves as a web of inter-relations for individual item detection has a close relation to the contextual cueing research developed by Chun (2000) and Chun and Jiang (1998). In their contextual cueing experiments they show how a specific item can be detected faster on repeated trials if the neighboring items of the target in the scene remain the same. We interpret these results not only as contextual information being present and helping target detection, but also as this contextual information being structurally tied to other information of the scene. In our view, relational information is the basis of why contextual cuing can occur. 

To summarize, we propose to extend Wheeler and Treisman’s (2002) concept of short-term visual memory as organized in parallel stores corresponding to particular feature dimensions (color, shape, size, etc.) by proposing the existence of relational information within each local item information unit in each feature store. Relational information may not be restricted to a given feature dimension but can be extended when binding of more than one feature type into an object is tested (Wheeler and Treisman, 2002). What we have shown is that it does not exist by default between dimensions but it does within an attended dimension. In this model the structural gist facilitates access to the different local information within a perceived scene. 

The fact that all local information is a composition of individual information and relational information gives a whole new face to the understanding of what can be called “an independent and unique feature” for perception. Even though we have shown the existence of relational information for visual short-term memory, we believe the idea could extend to perception in general. It has notable implications for cognitive neuroscience in that it allows different ways to code or represent a particular object, depending on the information present in the scene. The notion implies a widening of the search for neuronal correlates of feature and object information. It must consider this relational aspect in the coding of local information. 

Theoretical positions claiming a poor representation of the world by our visual system have already been suggested in the context of change blindness experiments (O’Regan, Rensink, & Clark, 1999; Rensink, O’Regan, & Clark, 1997; Rensink, 2002). We support such a position by claiming that instead of encoding all available information at the same individual level for later access, observers code only a very small portion as individual and accessible, namely that which is at the focus of attention. This is the information that observers have immediate cognitive access to and corresponds to the accessible item-content of visual short-term memory. Because local information is constituted by relational information, it is possible to have access to some aspects of global scene information by holding in memory only a few items of local, detailed information. We suggest that the remainder of the explicit information in the visual field is left in the visual world for further access, and so acts as an external memory store (O’Regan, 1992). However, we have to know how to obtain this information when needed, and so we use relational information contained in each unit in VSTM to know where to search in the real world for the individual information required. 

Acknowledgments

Authors would like to thank two anonymous referees for their helpful comments. Financial support was provided to JRV and HLG by PhD grants from the French government. 

Commercial relationships: none. 

Corresponding author: Juan R Vidal. 

Email: . 

Address: LENA UPR640, Hopital de la Salpetriere 47, Bd de l’Hopital, 75651 Paris, France.