The retention and disruption of color information in human short-term visual memory

Abstract

Previous studies have demonstrated that the retention of information in short-term visual perceptual memory can be disrupted by the presentation of masking stimuli during interstimulus intervals (ISIs) in delayed discrimination tasks (S. Magnussen & W. W. Greenlee, 1999). We have exploited this effect in order to determine to what extent short-term perceptual memory is selective for stimulus color. We employed a delayed hue discrimination paradigm to measure the fidelity with which color information was retained in short-term memory. The task required 5 color normal observers to discriminate between spatially non-overlapping colored reference and test stimuli that were temporally separated by an ISI of 5 s. The points of subjective equality (PSEs) on the resultant psychometric matching functions provided an index of performance. Measurements were made in the presence and absence of mask stimuli presented during the ISI, which varied in hue around the equiluminant plane in DKL color space. For all reference stimuli, we found a consistent mask-induced, hue-dependent shift in PSE compared to the “no mask” conditions. These shifts were found to be tuned in color space, only occurring for a range of mask hues that fell within bandwidths of 29–37 deg. Outside this range, masking stimuli had little or no effect on measured PSEs. The results demonstrate that memory masking for color exhibits selectivity similar to that which has already been demonstrated for other visual attributes. The relatively narrow tuning of these interference effects suggests that short-term perceptual memory for color is based on higher order, non-linear color coding.

Introduction

The ability of the brain to store sensory information in short-term memory provides an important means by which detail about the surrounding environment can be retained and subsequently used to mediate or direct behavior (Baddeley, 1986). This link between perception and memory has been further emphasized by studies that have highlighted the close association that exists between the neural mechanisms that mediate the storage of sensory information in short-term memory and those involved in its sensory encoding (Bisley & Pasternak, 2000; Fuster, 1997; Gibson & Maunsell, 1997; Graham, Barense, & Lee, 2010; Kang, Hong, Blake, & Woodman, 2011; Pasternak & Greenlee, 2005). This association has been most clearly articulated for our visual sense, where the retention of information relating to different attributes of visual stimuli has been proposed to occur within what has been termed “low-level perceptual memory” (Magnussen, 2009; Magnussen & Greenlee, 1999). This form of memory is described as operating in a non-declarative, implicit fashion and constitutes a pre-semantic level of storage for low-level sensory information (Magnussen, 2000, 2009). Low-level visual perceptual memory is thought to be based on neural activity that takes place within a network of brain areas distributed throughout the cerebral cortex that are located beyond V1 but early in the cortical processing stream (Fuster, 1997; Magnussen, 2009; Offen, Schluppeck, & Heeger, 2009). Related concepts include sensory working memory (Pasternak & Greenlee, 2005), the perceptual representation system (Schacter, Wagner, & Buckner, 2000), and weak or fragile forms of visual short-term memory (Sligte, Scholte, & Lamme, 2008, 2009). 

The short-term storage of visual information has been studied in detail at both behavioral and single neuronal levels (Bisley & Pasternak, 2000; Bisley, Zaksas, Droll, & Pasternak, 2004; Bisley, Zaksas, & Pasternak, 2001; Magnussen & Greenlee, 1992, 1999; Magnussen, Greenlee, Asplund, & Dyrnes, 1991; Magnussen, Greenlee, & Thomas, 1996; Miyashita & Chang, 1988; Pasternak & Zaksas, 2003; Regan, 1985; Zaksas, Bisley, & Pasternak, 2001). What is apparent from these studies is that short-term perceptual memory is dimension- or feature-specific, i.e., there appears to be a series of parallel mechanisms linked to memory formation that are devoted to particular attributes of a visual stimulus, such as its contrast, spatial frequency, or motion (Magnussen, 2000; Magnussen & Greenlee, 1999; Pasternak & Greenlee, 2005). Such feature specificity has been demonstrated by “dual-task” experiments where observers have to retain information and make subsequent judgments about two stimulus features. This can be done for different attributes (e.g., contrast and spatial frequency) virtually without impairment. However, when observers are asked to make judgments about the same feature (e.g. two spatial frequencies), thresholds are significantly elevated (Magnussen & Greenlee, 1999; Magnussen et al., 1991). Feature specificity in short-term perceptual memory has also been demonstrated by the effects of interference or masking stimuli presented during the ISIs of delayed discrimination tasks (Bennett & Cortese, 1996; Magnussen & Greenlee, 1992; Magnussen et al., 1991; McKeefry, Burton, & Vakrou, 2007). These so-called “memory masking” effects are outside the temporal range of those that are normally associated with traditional sensory masking paradigms (Breitmeyer, 1984). Experiments like these highlight the fact that sensory information that is retained by short-term visual perceptual memory is vulnerable to disruption or disturbance by subsequently presented visual stimuli. Crucially, this disruption occurs only for certain relevant features of the mask stimulus. Moreover, the effects are tuned, occurring only across narrow ranges of these features (Lalonde & Chaudhuri, 2002; Magnussen & Greenlee, 1992; Magnussen et al., 1991; McKeefry et al., 2007). These selective effects have been interpreted as revealing a modular organization of visual short-term memory that is purported to consist of an array of parallel stores, each tuned to a relatively narrow range of stimulus parameters, which are linked in a lateral inhibitory network, where interference occurs within but not between stores (Magnussen, 2000). 

Color forms an important feature of our visual environment from which information can be inferred. The sensory analysis of color is based on anatomically segregated and physiologically distinct processing pathways (see Gegenfurtner, 2003; Solomon & Lennie, 2007 for reviews). Subcortical color processing is based on outputs from L- (long), M- (middle), and S- (short) wavelength sensitive cones that interact in a linear fashion to form “red–green” (L–M) and “blue–yellow” (S–[L + M]) opponent mechanisms. This cone-opponent model of organization has satisfactorily accounted for many aspects of color perception (Derrington, Krauskopf, & Lennie, 1984; DeValois, Abramov, & Jacobs, 1966; Mullen & Losada, 1994). However, within the visual cortex, the neural processing of color appears to undergo a transformation. Numerous experimental observations, both behavioral and neurophysiological, point to the existence of more than two chromatic mechanisms with spectral sensitivities that are very different from the cone-opponent channels (Clifford, Spehar, Solomon, Martin, & Zaidi, 2003; Conway, 2003; DeValois, DeValois, & Mahon, 2000; DeValois, DeValois, Switkes, & Mahon, 1997; Goda & Fujii, 2001; Krauskopf, Williams, Mandler, & Brown, 1986; Lennie, Krauskopf, & Sclar, 1990; Li & Lennie, 1997; McGraw, McKeefry, Whitaker, & Vakrou, 2004; Webster & Mollon, 1991; Xiao, Casti, Xiao, & Kaplan, 2007; Zaidi & Halevy, 1993; see Eskew, 2009 for a review). One important difference is that these so-called “higher order” chromatic mechanisms have been shown to have narrower spectral tuning characteristics (DeValois, DeValois et al., 2000; Goda & Fujii, 2001; McKeefry, McGraw, Vakrou, & Whitaker, 2004; but see D'Zmura & Knoblauch, 1998 for a counterview), a property that can only arise as a result of non-linear combinations of cone inputs (DeValois, Cottaris, Elfar, Mahon, & Wilson, 2000; DeValois, DeValois et al., 2000). This property further differentiates them from cone-opponent mechanisms that are based on linear combinations of cone inputs and are more spectrally broadband (Derrington et al., 1984; DeValois, Cottaris et al., 2000). 

Up to now, there has been little attempt to try and examine short-term memory for color in the context of the underlying sensory physiology described above. This approach seems particularly pertinent in view of the recent emphasis on the close links between the neural mechanisms that underlie the analysis of sensory information and those involved in its retention in short-term memory (Pasternak & Greenlee, 2005). The aim of these experiments was to examine the extent to which perceptual memory displays selectivity for stimulus color. We used a memory masking paradigm in order to assess how the chromaticity of a masking stimulus determines the extent to which it can interfere with the fidelity of a stored representation of a colored reference stimulus. This paradigm has been successful in revealing the spatial tuning and speed selectivity of perceptual memory in previous experiments (Magnussen & Greenlee, 1992; Magnussen et al., 1991; McKeefry et al., 2007). There are two key questions we wanted to address: first, do parallel stores for stimulus color exist in short-term perceptual memory, similar to those that have been shown for spatial frequency and speed? Second, if memory masking experiments do reveal some form of color selectivity, can this tell us anything about how color information is organized in short-term perceptual memory? For example, is color information retained by mechanisms that are based on linear, broadband cone-opponent processing or instead stored by non-linear, narrowband, higher order color processing mechanisms in the brain? 

Methods

Stimuli

The color stimuli used in these experiments consisted of circular colored patches with sharp edges that subtended 1.5° and were presented on a uniform gray background [illuminant C (CIE 1931 chromaticity coordinates; x = 0.310, y = 0.316) of the same mean luminance (12.5 cd/m²)]. The stimuli were generated using purpose built software that drove a visual stimulus generator (VSG 2/5; Cambridge Research Systems (CRS), Rochester, UK) and were displayed on a high-resolution color graphics monitor (GDM520, Sony, Tokyo, Japan, frame rate of 120 Hz) that subtended 26.3° × 32.75°. In the main experiments, the chromaticities of the color stimuli were specified as equal length vectors in CIE 1931 color space that were defined by their angle of rotation (ϕ) in the isoluminant plane (see Figure 1). The endpoints of these vectors formed a circle around illuminant C, and chromaticity coordinates for twenty of the main color stimuli used to define this circle are given in Table 1. The cone contrasts produced by these stimuli on the background were calculated using the Judd modified values (x′, y′, Y′) values. X′, Y′, Z′ tristimulus values were then used in conjunction with the Smith and Pokorny (1975) cone fundamentals to obtain the magnitude of cone excitation for each component color from which modulation for each cone (L_c, M_c, and S_c) was calculated. Similar to the DKL color space (Derrington et al., 1984), the cardinal [L–M] (ϕ = 0–180°) and S–[L + M] (ϕ = 90–270°) axes were defined as those axes along which the stimuli elicited zero S-cone modulation or zero L- and M-cone modulation, respectively. Monitor calibration was performed by using a ColorCal probe (Cambridge Research Systems, Rochester, UK) and with a Spectrascan PR-650 SpectraColorimeter (Photoresearch, Chatsworth, California, USA). In order to ensure the accuracy of the chromaticity coordinates delivered by the CRS software, we performed additional calibration procedures as described in Parry, McKeefry, and Murray (2006). 

Figure 1

 

Specification of color stimuli used in the experiments. The left-hand panel shows DKL color space that was used to specify the chromatic stimuli used in these experiments. The right-hand panel illustrates an enlarged section of the CIE 1931 chromaticity diagram to show the direction of movement through color space when hue and saturation are altered. Color stimuli were generated by equal length vectors in this space with the same point of origin, illuminant C (CIE 1931 x = 0.310, y = 0.316). Color stimuli were sampled around the equiluminant plane with their hue being defined by the angle of rotation (ϕ). The concentric circles indicate the baseline saturation of unity together with higher and lower saturations (1.5 and 0.5). Saturation of the stimulus could be changed by increasing the vector length in the direction shown.

Table 1

 

CIE 1931 xy chromaticity coordinates of 20 of the color stimuli that defined the hue circle employed in the color memory masking experiments. In all cases, Y = 12.5 cd/m².

Stimulus axis (ϕ)	x	y
0	0.385	0.29
18	0.373	0.268
36	0.36	0.251
54	0.333	0.24
72	0.309	0.237
90	0.284	0.241
108	0.262	0.253
126	0.245	0.27
144	0.234	0.293
162	0.231	0.317
180	0.235	0.342
198	0.247	0.364
216	0.264	0.381
234	0.287	0.392
252	0.311	0.395
270	0.336	0.391
288	0.358	0.379
306	0.375	0.362
324	0.386	0.339
342	0.389	0.315
360	0.385	0.29

In additional experiments, the luminance contrast content of the mask stimuli was varied via manipulation of their luminance ratio (LR) defined as 

$\mathrm{L}\mathrm{R}=\frac{L_{\mathrm{m}\mathrm{s}}}{L_{\mathrm{m}\mathrm{s}}+L_{\mathrm{b}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{d}}},$

(1)

where L _ms = luminance of masking stimulus and L _bkgrd = luminance of background. An LR = 0.5 generates a mask stimulus that is photometrically equiluminant with the background (i.e., contains only chromatic contrast). Values either side of this generate stimuli containing varying amounts of luminance and chromatic contrast, with luminance increments denoted by LR > 0.5 and decrements by LR < 0.5. 

Four main reference stimuli were used in this set of experiments, which were reported by the individual observers as being exemplars of four main color categories red, green, blue, and yellow, i.e., unique hues. These stimuli were specified by preliminary experiments where observers performed a hue categorization (naming) procedure in order to determine their location in DKL color space. A 4-alternative forced-choice procedure was used where they indicated whether presented color stimuli appeared blue, green, yellow, or red (DeValois et al., 1997; Parry et al., 2006). We presented 20 different chromatic axes that represented vectors that were equally spaced in steps of 18° ranging from ϕ = 0° to ϕ = 360° around illuminant C. The results were analyzed by deriving four color naming functions p[red], p[blue], p[green], and p[yellow], where p[color] was the proportion of times that a particular test hue was called that color out of a total of 20 presentations. The exemplars of four main hues (red, blue, green, and yellow) were defined as the central maxima of the hue scaling functions. In control experiments, we also used reference stimuli that were equally likely to be classified into adjacent color categories rather than unique or exemplar hues. Results obtained with these “non-unique” stimuli were similar to those obtained with the standard reference stimuli. 

Procedure

In the main experiments, a delayed color discrimination paradigm was used to measure the fidelity of stored color information in perceptual memory. The paradigm employed a 2-alternative forced-choice procedure in conjunction with a method of constant stimuli. Each trial began with the presentation of a reference stimulus of 380 ms duration; this was followed by a 5 s ISI, and in the middle of this period, a mask stimulus of variable chromaticity was presented for 2 s. At the end of the ISI, a test stimulus was presented for 380 ms (see Figure 2). The reference, mask, and test stimuli were separated horizontally to avoid retinal adaptation effects. In training sessions, the observers were also instructed to maintain fixation on a centrally placed cross throughout the trial. The reference was presented 3° (center to center) to the right of the fixation point, the mask stimulus was centered on the fixation point, and the test was presented 3° (center to center) to the left of fixation. The test stimulus could be one of seven colors that sampled equally (in terms of ϕ) across a range of hues on the isoluminant color circle either side of the reference stimulus color. The range of test stimuli used was determined in preliminary experiments so that the two endpoint colors were those hues closest to the reference stimuli that were 100% discriminable from it. This ensured that the test and reference stimuli fell within the same color categories rendering any attempt by observers to employ verbal, color naming strategies ineffective for task performance. 

Figure 2

 

Schematic representation of the delayed color discrimination task. In the baseline condition, a reference stimulus appeared 3° to the right of a central fixation cross for 380 ms. Following an interstimulus interval (ISI) of 5 s, a test stimulus appeared 3° to the left of the fixation mark also for 380 ms. Observers were requested to indicate, via a button press, how they perceived the color of the test relative to the reference (see text). In the memory masking condition, the paradigm was identical except that an additional mask stimulus appeared for 2 s during the ISI, appearing in the center of the screen coincident with the fixation point. All color stimuli subtended 1.5°.

Following the end of each trial, the observers were instructed to respond by button press (CB3 response box; Cambridge Research Systems) to indicate where they considered the test stimulus hue to be located on the color circle relative to that of the remembered reference color. Observers were given prior training on the relationship of colors around the hue circle in preliminary training sessions. In practice, this meant that for a blue reference stimulus, for example, observers were instructed to indicate whether the test stimulus appeared to be more “green” or “purple,” indicating a clockwise or anti-clockwise shift around the hue circle, respectively. For a yellow reference stimulus, the response was either more “orange” or “green” (anti-clockwise rotation) and so forth for each of the reference stimuli used. This procedure enabled us to plot psychometric functions that then allowed us to assess the effects of different mask stimuli on performance, in particular the extent to which mask stimuli affected the point of subjective equality (PSE) between the test and remembered reference colors. Performance on the delayed color discrimination paradigm was assessed relative to performance on the baseline condition, where no mask stimulus was introduced during the ISI. Each psychometric curve was based on a minimum of 140 trials that were randomly interleaved across four 1 h sessions. This randomized presentation was adopted in order to prevent the build up of more long-term representations in memory of the stimuli. 

Data analysis

The psychometric data were fitted by a logistic function of the form: 

$y=\frac{100}{(1+e^{\frac{(\varphi -\mu )}{\theta }})},$

(2)

where y is the percentage of times the test stimulus was reported as being rotated in an anti-clockwise rotation in color space relative to the reference, ϕ is the chromatic axis of the test stimulus, μ is the relative rotation in color space corresponding to the 50% level on the psychometric function (i.e., the point of subjective equality (PSE)), and θ is an estimate of the color discrimination threshold. 

The PSE data were plotted as a function of mask chromatic axis and fitted by a first derivative of a Gaussian function described by the following equation: 

$y=y_{\mathrm{p}\mathrm{o}\mathrm{s}}+[\left(\frac{A}{(\sigma .e^{-0.5})}\right)*(\varphi -x_{\mathrm{p}\mathrm{o}\mathrm{s}})*e^{-\left(\frac{(\varphi -x_{\mathrm{p}\mathrm{o}\mathrm{s}}{)}^2}{2{\sigma }^2}\right)}],$

(3)

where y is the point of subjective equality (PSE), ϕ is the chromatic axis of the mask in DKL color space, σ is the standard deviation of the Gaussian, A is the half-amplitude of the function, and x _pos, y _pos is the origin of the function (when ϕ = x _pos, PSE = y _pos). The half-amplitude of this function represents the magnitude by which the PSE deviates from baseline. The maxima and minima of this function occur at mask chromatic axis orientations ±σ units from the origin (i.e., (ϕ − x _pos) = ±σ). 

Observers

Five observers took part in the study (3 females and 2 males; mean age = 36.2 years, SD = 5.6 years), two of whom were authors, the remaining three were naive as to the aims of the experiment. All gave informed consent and had normal color vision according to Rayleigh matches made on an HMC Anomaloskop and the Farnsworth-Munsell 100-Hue test. All had 6/6 or corrected to 6/6 Snellen visual acuity. The experiments were performed in a darkened room and observers fixated on a small black cross on the center of the screen, which was viewed binocularly from a distance of 114 cm. 

Results

Figure 3 shows representative individual data obtained from the memory masking experiments for each of the four main reference color stimuli (red, green, blue, and yellow). The graphs plot how PSEs vary as a function of mask stimulus chromatic axis (ϕ) and are specified in terms of the rotation of the chromatic axis in DKL color space of the matched test stimulus relative to the reference stimulus. As can be seen in the plots, the color matches made by the observers are highly dependent on the chromaticity of the masking stimulus and a similar pattern can be observed regardless of the color of the reference stimulus. When mask color is identical to the reference (i.e., zero relative rotation), the resultant PSEs are similar to those obtained for the baseline (no mask) conditions (horizontal dashed lines). However, when the chromatic axis of the mask stimulus shifts away from that of the reference, there are small but systematic variations in the PSE away from baseline levels. For example, when the mask axis rotates in a clockwise direction in color space (positive relative rotation), the matches made by the observers are shifted away from the baseline value toward the hue of the mask stimulus. The deviation from baseline reaches a maximum with increasing rotation but then starts to decrease and approach baseline levels as mask chromaticity shifts even further away from that of the reference. When the mask chromatic axis rotates away from the reference in an anti-clockwise direction (negative relative rotations), the PSEs are shifted in the opposite direction, reaching a maximum then returning to baseline. 

Figure 3

 

Data from the color memory masking experiment. Representative results are shown for single observers for four reference colors (blue (ϕ = 126°), red (ϕ = 18°), green (ϕ = 216°), and yellow (ϕ = 288°)). PSEs obtained from the psychometric curve fits are plotted as a function of rotation of mask chromatic axis relative to the reference in color space. The horizontal dashed lines indicate baseline (i.e., no mask) performance and the vertical dashed lines indicate conditions where the reference and the mask stimuli have the same chromatic axis.

The pattern of these results suggests that the mask stimulus can interfere with the stored representation of the color of the reference stimulus and can induce a shift in the point of perceived equality as long as the mask color is different, but not too different, from the reference stimulus. Thus, it would appear that memory masking for color exhibits tuning or selectivity similar to that which has already been demonstrated for other visual attributes such as motion and spatial frequency (Magnussen et al., 1991; McKeefry et al., 2007). In order to ensure that these masking effects are indeed localized to a specific region of color space, we performed an additional experiment on two subjects in which delayed color matching was measured in the presence of mask stimuli sampled from across the full 360° of the color circle. The results from this experiment are shown in Figure 4 where the effects of masking can be seen to occur only within a narrow range centered on the reference stimulus (blue ϕ = 120°). 

Figure 4

 

Data from a control experiment where mask stimuli were sampled from across the full 360° of DKL color space. In this example, the reference stimulus was blue (ϕ = 126°) and the results are for a single observer (DM).

The tuned nature of these masking effects prompted us to fit the data with first derivative of Gaussian functions that allowed us to derive an estimate for the bandwidth (σ) of these tuning functions (see Methods section). The results of this procedure are shown in Figure 5 where the group-averaged data (n = 5) for each reference stimulus have been fitted by these functions. The resulting values for the bandwidths of the masking effects were: 36.8° for the blue reference stimulus, 33.4° for the green, 29.1° for the yellow, and 34.1° for the red. 

Figure 5

 

Group-averaged memory masking data (n = 5) for the four main reference stimuli. The data have been fitted with the 1st derivative of a Gaussian function (see Methods section). From this fitting procedure, the value σ provides an estimate of bandwidth in DKL color space across which masking stimuli are effective in interfering with the retention of the chromatic stimulus in short-term perceptual memory. Error bars represent ±1 SD of the mean.

Implicit in our stimulus design is the assumption that equal length vectors in color space generate stimuli that have the same perceived saturation (see Figure 1b). This is not a valid assumption in view of the perceptual non-uniformity of this color space. In order to demonstrate that these color memory masking effects were not simply the result of differences in perceived saturation of the masking stimuli, we examined the effects of varying their saturation. This was achieved by varying the vector length in color space and was expressed relative to the standard condition (see Figure 1b). We chose two masking stimuli; one was an effective mask, i.e., a mask stimulus that induced a shift in PSE away from baseline performance. The other was an ineffective mask defined by a chromatic axis that lay well outside the region of color space where it had any measurable effect on the PSE obtained for the chosen reference color. Figure 6 shows results from 2 observers who performed this experiment. The reference stimulus, in this case, was an exemplar of the red color category and the effective mask had a chromatic axis (ϕ) that was rotated 34° clockwise in color space relative to the reference. The ineffective mask had a chromatic axis rotated 108° clockwise, relative to the reference. As can be observed, ineffective masks have minimal effect on the PSE obtained from the delayed discrimination experiments and this is the case regardless of their saturation. An ineffective mask cannot be made effective simply by increasing its saturation. By comparison, over the same range, an effective mask generated increasingly larger shifts in PSE with increasing saturation. 

Figure 6

 

PSE values plotted as a function of mask saturation levels. The graphs plot how PSEs for a red reference stimulus (ϕ = 18°) are shifted for effective masks (ϕ = 34°) of increasing saturation and ineffective masks (ϕ = 108°). The data represent the average performance of 2 observers and have been fitted by linear regression lines. The horizontal dashed line indicates baseline (no mask) performance.

A key property of visual perceptual memory is that it is attribute or dimension specific (Magnussen, 2009). This has been demonstrated in memory masking experiments by the fact that masking only occurs for certain relevant parameters (Magnussen & Greenlee, 1999). We wanted to examine in an analogous fashion to what extent the introduction of other non-color stimulus attributes had on the effectivity of chromatic masking stimuli. Previous studies of visual perceptual memory point to the involvement of separate systems in the retention of luminance and color information (Sachtler & Zaidi, 1992; Yoshizawa, Kubota, & Kawahara, 2011). Therefore, in an additional experiment, we investigated the influence of adding luminance contrast to the chromatic masking stimuli. In the experiments described in this study so far, the mask, reference, and test stimuli have all been equiluminant with the background. If the retention of chromatic and luminance contrast information does indeed occur within separate perceptual memory mechanisms, then the prediction would be that the addition of luminance contrast to the masking stimulus should reduce its ability to interfere with the retention of color information, i.e., the shifts in PSE should be reduced. To test this, we repeated the delayed color discrimination experiments using masking stimuli that varied in terms of their luminance contrast via manipulation of their luminance ratio (LR; see Methods section). The data in Figure 7 show how this affects measured PSEs and demonstrate that the greatest shift occurs when the mask, like the reference and test stimuli, is isoluminant (LR = 0.5) and contains no luminance contrast. When luminance contrast is added to the mask and LR increases or decreases from a value = 0.5, PSEs fall to levels similar to those obtained for the no mask condition. Thus, in line with the prediction, the addition of luminance contrast to masking stimuli renders them less effective in their ability to interfere with the retention of chromatic information. This is consistent with the idea that chromatic and luminance contrasts are stored separately within short-term perceptual memory. 

Figure 7

 

Memory masking as a function of luminance ratio. PSEs are plotted as a function of mask luminance ratio. The vertical dashed line represents the luminance ratio of the reference and test stimuli; as the luminance ratio tends toward zero, the mask becomes a luminance contrast decrement stimulus. Toward a luminance ratio of 1, the mask becomes a luminance contrast increment. The data represent the average performance of 2 observers, and the horizontal dashed line indicates baseline (no mask) performance.

Discussion

In this study, we have demonstrated using a memory masking paradigm the existence of selective, hue-dependent, mask-induced interference effects on stored representations of color stimuli within short-term visual perceptual memory. These findings suggest the existence of a short-term perceptual memory system that can retain sensory information about stimulus color that comprises an array of stores that retain chromatic information across limited bandwidths of color space. These properties are consistent with previous studies that have demonstrated similar organizational principles for perceptual memory for other visual attributes such as spatial frequency and motion (Bennett & Cortese, 1996; Lalonde & Chaudhuri, 2002; Magnussen et al., 1991; McKeefry et al., 2007). 

The existence of a perceptual or sensory memory system for the short-term retention of color information mirrors the extent to which color has its own anatomically and physiologically distinctive processing pathway from the very earliest stages of visual processing. A number of previous studies have highlighted the fact that information about the color of stimulus can be stored independently of other stimulus attributes, such as pattern and luminance (Cornelisson & Greenlee, 2000; Magnussen, Greenlee, & Thomas, 1996; Nilsson & Nelson, 1981; Sachtler & Zaidi, 1992; Stefurak & Boynton, 1986; Yoshizawa et al., 2011). Furthermore, humans are able to retain information about color within short-term memory with a high degree of accuracy over relatively long periods of time. However, color memory is not perfect and when human subjects view a color and then try to match it from memory after a period of time has elapsed, there are often slight, but measurable, differences between the original and memory-matched color in terms of hue, saturation, and brightness (Bartleson, 1960; Burnham & Clark, 1954, 1955; Collins, 1931; Jin & Shevell, 1996; Newhall, Burnham, & Clark, 1957; Nilsson & Nelson, 1981; Pérez-Carpinell, Baldovi, de Fez, & Castro, 1998; Siple & Springer, 1983). More long-term memory representations of color have also been shown to play an important role in the mechanisms that underpin color constancy (Jin & Shevell, 1996; Ling & Hurlbert, 2008). In addition, there are the intriguing effects that have been demonstrated on low-level visual perception mediated by what have been termed “memory colors” (Hansen, Olkkonen, Walter, & Gegenfurtner, 2006)—a term that refers to the fact that certain colors have a very close association with specific objects and are often integral to their identity. 

While the retention of color information in short-term perceptual memory undoubtedly provides us with a great deal of information about objects and surfaces within our environment, what do the experimental results reported here tell us about how this information is organized in short-term perceptual memory? If we look at results from the spatial frequency domain, the bandwidths of tuning revealed by memory masking experiments have been found to be of the order of approximately ±1 octave (Magnussen et al., 1991; Nemes, Whitaker, Heron, & McKeefry, 2011). This value is remarkably similar to estimates of bandwidths for spatial frequency channels that exist in low-level sensory visual processing, revealed by sensory masking and contrast adaptation studies (Blakemore & Campbell, 1969; Blakemore & Nachmias, 1971; Blakemore, Nachmias, & Sutton, 1970; Campbell & Robson, 1968; Georgeson & Harris, 1984). This correspondence has been taken as evidence to support the idea that there is close association between the sensory mechanisms that are involved in the low-level visual processing of spatial frequency and those that are involved in the storage of this information in perceptual or sensory memory (see Pasternak & Greenlee, 2005 for a review). Following the same rationale, can the tuning revealed by the color memory masking experiments in this study be linked with the sensory processing of color in the visual system? We know that, following photon capture by the retinal photoreceptors, color information is signaled by cone-opponent mechanisms (DeValois et al., 1966). These mechanisms rely upon linear combinations of L-, M-, and S-cone inputs and predominate in the subcortical and early stages of cortical color processing. Cone-opponent mechanisms exhibit responses to color stimuli that vary sinusoidally across DKL color space (Derrington et al., 1984; DeValois, Cottaris et al., 2000). Figure 8a plots data recorded from such a cone-opponent (−L + M) neuron recorded from the monkey LGN (DeValois, DeValois et al., 2000). Its response is plotted as a function of chromatic axis in DKL color space and has been fitted by a Gaussian function that provides an estimate of bandwidth (σ). This gives a value of approximately 60°, typical of the bandwidth estimates for linear chromatic mechanisms found in subcortical and early V1 color processing (DeValois, Cottaris et al., 2000; D'Zmura & Knoblauch, 1998). This bandwidth, however, is considerably wider than those revealed in our memory masking experiments. Therefore, it seems unlikely that broadly tuned, linear cone-opponent mechanisms mediate these interference effects. An alternative basis for these effects might lie in the fact that rather than adhering to two cone-opponent or cardinal mechanisms, color processing within the cerebral cortex instead relies upon multiple “higher order” chromatic mechanisms that are tuned to many different directions in color space (Clifford et al., 2003; DeValois, Cottaris et al., 2000; DeValois, DeValois et al., 2000; DeValois et al., 1997; Goda & Fujii, 2001; Krauskopf et al., 1986; Li & Lennie, 1997; McGraw et al., 2004; Webster & Mollon, 1991; Zaidi & Halevy, 1993). These higher order mechanisms appear to be the result of recombinations of outputs from the cone-opponent mechanisms (see Eskew, 2009 for a review). Currently, there is some debate as to whether they arise as the result of linear or non-linear interactions. Close to threshold, higher order mechanisms have been found to be largely linear with bandwidths of approximately 60°, similar to those exhibited by cone-opponent mechanisms (D'Zmura & Knoblauch, 1998; Giulianini & Eskew, 1998; Hansen & Gegenfurtner, 2006; Sankeralli & Mullen, 1997). On the other hand, at suprathreshold levels, there is evidence to suggest varying degrees of non-linearity in the formation of these higher order chromatic mechanisms, which leads to the generation of more narrowly tuned color mechanisms with bandwidths of the order of 30–40° (Clifford et al., 2003; Goda & Fujii, 2001; McKeefry et al., 2004). These values are in closer accord with tuning characteristics revealed by the memory masking experiments in this study. Thus, we might speculate that the chromatic information utilized by short-term perceptual memory is derived from a stage in color processing beyond that where the transformation from linear, broadband, cone-opponent processing to non-linear, more narrowband higher order chromatic processing has taken place. Narrowly tuned chromatic mechanisms are significant in color processing in that they provide a potential link with specific hues—something that broadband mechanisms do not offer. In order for more broadly tuned mechanisms to signal stimulus color, it would require another stage at which their outputs could be compared (Eskew, 2009). Outputs from narrowly tuned chromatic mechanisms, on the other hand, could directly signal stimulus hue. This link raises the possibility that color information in short-term perceptual memory is organized around perceptual color categories. A similar suggestion has been made in the light of experimental findings that have demonstrated that the extent of degradation in the fidelity of remembered colors is less marked for more perceptually relevant or focal colors (Berlin & Kay, 1969; Heider, 1972; Nemes, Parry, & McKeefry, 2010). Also consistent with this idea is the fact that hue naming functions, which were used in preliminary experiments in this study to define the reference stimuli, have bandwidths that are similar to those obtained from the color memory masking experiments. In Figure 8b, blue (p[blue]) and yellow (p[yellow]) hue naming functions have been plotted as a function of chromatic axis. The values of σ obtained from the Gaussian fits reveal bandwidths of 41.8° for blue and 29.1° for yellow hue naming functions, comparable not only to those values revealed by color memory masking in this study but also to those reported for non-linear, higher order chromatic mechanisms. Thus, there is some circumstantial evidence to suggest that color categories may form the basis for the storage of chromatic information in perceptual memory. However, there were discrepancies in terms of the bandwidth estimates between the memory masking and the hue naming data in the case of red and green stimuli. The hue naming functions (not shown) for red (p[red]) and green (p[green]) are much broader, exhibiting plateaus across a range of color space (see also DeValois et al., 1997). This discrepancy may be a consequence of the fact that the hue naming method we employed restricted observer responses to only four basic color categories (red, green, blue, and yellow) to describe the stimuli. There is the possibility, therefore, that these broader categories might consist of further subcategories with narrower bandwidths that may be more in keeping with the results revealed by memory masking. Certainly, further work will be required to establish more rigorously whether perceptual color categories form the basis around which short-term perceptual memory for color is organized. 

Figure 8

 

(a) Responses of a cone-opponent (−L + M) LGN neuron plotted as a function of chromatic axis in DKL color space. The data have been taken from DeValois, DeValois et al. (2000) and have been normalized and fitted with a Gaussian function, the bandwidth (i.e., the standard deviation) of which is specified by the parameter σ. (b) Data taken from the hue naming experiment used in this study to define reference stimuli. The data points represent the proportion of times that color stimuli were identified as, in these examples, blue (p[blue]) and yellow (p[yellow]) as a function of chromatic axis (ϕ). Stimulus color here is plotted in terms of relative rotation from the axis that defined the unique or exemplar hue. In the case of the blue color naming data, 0 refers to ϕ = 126° in DKL color space and ϕ = 286° in the case of the yellow data. The functions represent the average data from 5 observers. As with the single-unit data shown in (a), these behavioral data have been fitted with a Gaussian function in order to obtain bandwidth estimates (σ).

Experimental data from the primate visual system has not, as yet, provided us with an unequivocal answer as to where in the cortex the transformation or reorganization from cone-opponent to perceptual color coding might occur. At the single-unit level, studies have demonstrated that many cortical chromatic neurons exhibit color tuning that is narrower than that found in the subcortical visual pathway (DeValois, Cottaris et al., 2000; Lennie et al., 1990; Thorell, DeValois, & Albrecht, 1984; Vautin & Dow, 1985). The emergence of neurons that respond to specific perceptual color categories appears to occur right from the level of V1 (Xiao et al., 2007) and they become more prominent in visual areas V2, V4, and posterior inferotemporal cortex (Conway, Moeller, & Tsao, 2007; Kiper, Fenstermaker, & Gegenfurtner, 1997; Komatsu, Ideura, Kaji, & Yamane, 1992; Stoughton & Conway, 2008; Xiao, Wang, & Felleman, 2003; Zeki, 1980). In the human brain, recent neuroimaging studies have indicated that the transformation from cone-opponent to more perceptually based color processing occurs at the level of visual areas V01 (ventral occipital) and V4 (Brouwer & Heeger, 2009). We suggest that these areas may constitute a possible locus for the neural activity that underpins short-term perceptual color memory. Certainly, area V4 has long been viewed (though not unanimously) as playing an important role in color processing (Brewer, Liu, Wade, & Wandell, 2005; Lueck et al., 1989; McKeefry & Zeki, 1997; Wade, Brewer, Rieger, & Wandell, 2002; Zeki, 1980). Its involvement in color perceptual memory, along with neighboring areas in the ventral extrastriate cortex, would be consistent with perceptual memory being based on neural activity based within a network of brain areas located beyond V1 but relatively early in the cortical processing stream (Fuster, 1997; Magnussen, 2009; Offen et al., 2009). 

In summary, this study has examined the retention of chromatic information by short-term visual perceptual memory. We have shown that the storage of chromatic information in this store is vulnerable to interference by the subsequent presentation of other chromatic stimuli. However, this interference is selective and is induced by stimuli that fall only within a relatively narrow range of color space. These effects point to the involvement of narrowly tuned, non-linear, higher order chromatic mechanisms as the basis for the retention of color information in short-term perceptual memory. 

Acknowledgments

V.N. was supported by a Ph.D. studentship funded by the Federation of Ophthalmic and Dispensing Opticians. N.R.A.P. is supported by the NIHR Manchester Biomedical Research Centre. 

Commercial relationships: none. 

Corresponding author: Declan J. McKeefry. 

Email: [e-mail]. 

Address: Bradford School of Optometry and Vision Science, University of Bradford, Richmond Road, Bradford BD7 1DP, UK.