Saturation and brightness modulate the effect of depth on visual working memory

Abstract

Although previous studies show inconsistent results regarding the effect of depth perception on visual working memory (VWM), a recent finding shows that perceptually closer-in-depth items are better remembered than farther items when combining the congruent disparity and relative size cues. In this study, we employed a similar change detection paradigm to investigate the effects of saturation and brightness, alone or in combination with binocular disparity, on VWM. By varying the appearance of the memory items, we aimed to manipulate the visual salience as well as to simulate the aerial perspective cue that induces depth perception. We found that the change detection accuracy was significantly improved for brighter and more saturated items, but not for items solely with higher saturation. Additionally, combining saturation with the congruent disparity cue significantly improved memory performance for perceptually closer items over farther items. Conflicting the disparity cue with saturation eliminated the memory benefit for the closer items. These results indicate that saturation and brightness could modulate the effect of depth on VWM, and both visual salience and depth perception affect VWM possibly through a common underlying mechanism of setting priority for attentional selection.

Introduction

Working memory is often considered to be a temporary storage system for holding and manipulating information. It serves as a link between sensory encoding and higher cognitive functions, thus is crucial for perception and cognition (Baddeley, 2007). It is also suggested to be correlated with fluid intelligence (Fukuda, Vogel, Mayr, & Awh, 2010; Unsworth, Fukuda, Awh, & Vogel, 2015). For the recent two decades, working memory related to visual information processing has gained growing interest and been studied extensively. 

Visual working memory (VWM) is found to be severely limited in capacity (Awh, Barton, & Vogel, 2007; Cowan, 2001; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Zhang & Luck, 2008). Research shows that no more than four items can be retained in VWM despite differences among the tested objects—for example, memory capacity is found to be about four for colored squares and less than two for shaded cubes (Alvarez & Cavanagh, 2004). However, people have not been aware of such a memory limitation until VWM was scientifically tested in laboratories. Indeed, we are rarely aware of any difficulty in temporarily maintaining and processing visual information in reality. It is presumed that long-term memory and other cognitive factors contribute in overcoming the capacity limitation of working memory (Brady, Störmer, & Alvarez, 2016). For example, certain objects may be prioritized for processing through attentional selection, yielding better memory for these objects. Research suggests that a space-based attentional selection could prioritize objects near the location of interest (Eriksen & Hoffman, 1973; Posner, 1980), and a feature-based attentional selection could prioritize objects sharing the same feature with the object of interest (Duncan, 1984; Qian, Zhang, Lei, & Han, 2018). In addition, studies show that VWM could be enhanced if visual information is perceptually grouped into chunks (Jiang, Chun, & Olson, 2004; Li, Qian, & Liang, in press; Xu, 2006). Gestalt principles, such as proximity, similarity, and connectedness, have been systematically tested and found to increase the capacity of VWM (Peterson & Berryhill, 2013; Woodman, Vecera, & Luck, 2003). 

The majority of research on VWM presented visual stimuli on a flat screen perpendicular to the line of sight. However, several studies employing stereoscopic depth have shown intriguing findings regarding the role of depth perception in VWM. Xu and Nakayama (2007) reported a small increase in the capacity of visual short-term memory if the visual stimuli were presented on two disparity-defined surfaces compared to those presented on a single surface. The authors presented the stereoscopic surfaces sequentially, because in their pilot study, simultaneous presentation of the surfaces did not affect memory performance. They also stressed that it is three-dimensional (3-D) surface, not depth information per se, that influences the performance. However, Reeves and Lei (2014) found that stereoscopic depth did not affect iconic memory performance by employing a partial report paradigm. In their study, a 3 × 3 letter array was briefly presented with each row located at one disparity-defined depth plane, and the observers were instructed to report one row of letters indicated by an arrow cue. The authors suggested that the divergence in these results could be due to participants successively switching their attention between the two 3-D surfaces in Xu's study, but they could not use such a strategy in their study. 

Although neither of the above two studies found a beneficial effect of stereoscopic depth on VWM using simultaneous presentation of stimuli, our recent study suggests differently by using an adapted change detection paradigm (Qian, Li, Wang, Liu, & Lei, 2017). We found that presenting the items in stereoscopic depth alone hardly affected VWM performance, which is consistent with the previous studies. However, when combining disparity and relative size depth cues, the memory performance was significantly better for perceptually closer-in-depth items than for farther items, despite that the overall performance for the two-depth-plane layout was not significantly different from the one-plane layout. The results indicate that depth perception affects VWM by enhancing the memory for closer-in-depth objects. In addition, it seems that this effect cannot be attributed to grouping by depth, but rather reflects modulation by attention. 

Binocular disparity and relative size cues were employed in our previous study to create depth perception. Disparity cue is usually more effective at near distances (Campbell, 1957; Hillis, Watt, Landy, & Banks, 2004; Ono & Comerford, 1977), but sometimes can be overridden by contradictory long-range monocular depth cues even at close distances (O'Leary & Wallach, 1980; Wallach & Zuckerman, 1963). Relative size has long been known as a powerful monocular depth cue and has also been extensively studied in the literature (Fitzpatrick, Pasnak, & Tyer, 1982; Ittelson, 1951; Marotta & Goodale, 2001). However, it is unclear whether other monocular depth cues could result in a similar depth effect on VWM. One of the monocular cues that has been found to induce depth perception is the so-called aerial perspective. Aerial perspective emerges from the fact that the air is filled with light-absorbing and light-scattering particles even on the clearest of days (Coren, Ward, & Enns, 2004). The lights from more distant objects must travel through the atmosphere for a greater distance and are subject to increased absorption or scattering by the particles in the air. Therefore, a distant object may appear to be slightly grayer or less pronounced than a nearer object that is physically of the same color (Ross & Plug, 1998). It is also referred to as relative brightness or relative contrast (Coren et al., 2004), because a more distant object may appear to be less bright, or lower in contrast. The brighter of the two otherwise identical objects tends to be judged as closer in absence of other cues; even when other cues are available, reduced contrast is associated with judging an object as more distant (Dresp-Langley, Durand, & Grossberg, 2002; Dresp-Langley & Reeves, 2012; O'Shea, Blackburn, & Ono, 1994; Rohaly & Wilson, 1999). Furthermore, research found that the perceived depth separation increased with increasing difference in brightness (Egusa, 1977) and saturation (Egusa, 1983). However, the effect of saturation on depth perception seems to be modulated by other factors—for example, hue (Egusa, 1983) and contrast polarity with respect to background (Dresp-Langley & Reeves, 2014). 

In the present study, we aimed to investigate the effects of saturation and brightness on VWM. Potentially, their effects on VWM may stem from two distinct underlying mechanisms. On the one hand, saturation and brightness can be manipulated to simulate the aerial perspective cue and therefore to induce depth perception. In this case, relative saturation and brightness work as monocular depth cues. By presenting these monocular depth cues either alone, or in conjunction with a congruent or incongruent binocular disparity cue, we are able to test whether and how these depth cues interact with each other and therefore affect the VWM performance. Similar to our previous study (Qian et al., 2017), we would expect that memory performance could be improved for perceptually closer items, as long as the depth cues create salient and stable depth perception. On the other hand, items differing in saturation and/or brightness may also differ in perceptual saliency. In this case, items with high saturation/brightness may benefit from selective attention and be prioritized for processing, leading to better memory performance without invoking any mechanisms of depth perception. Research suggests that visual salience could improve working memory performance by obtaining attentional priority in the encoding stage (Fine & Minnery, 2009; Melcher & Piazza, 2011; Pedale & Santangelo, 2015). In addition, changes in items with low saturation/brightness might be harder to detect, resulting in a more difficult comparison process between the memory item and the probe. Indeed, research showed that memory performance degraded with increasing difficulty in the decision stage of VWM (Lin & Luck, 2009). These would predict improved accuracy for high brightness/saturation items irrespective of their locations in depth. 

It is unclear which of the two mechanisms, either or both, are employed by the visual system. To our knowledge, few studies in the past literature have explored the effects of saturation and brightness on VWM by employing a change detection paradigm. However, there is one study— Olson and Jiang (2002) tested the memory performance for high- and low-saturation colors in separate blocks, and found that the effect of saturation approached, but did not reach, significance (p = 0.055, Experiment 2). This suggests that varying saturation alone may be insufficient, or difficult at least, to affect the VWM performance. Therefore, here we first hypothesized that saturation and brightness affect VWM through an underlying mechanism of depth perception, and employed a change detection paradigm similar to our previous study (Qian et al., 2017). Experiment 1 examined the joint effect of relative saturation and brightness, and Experiments 2 and 3 tested the individual effects of relative saturation and relative brightness, respectively. In addition, disparity cue was applied in all experiments to provide stereoscopic depth, whose direction was either consistent or inconsistent with that implied by relative saturation/brightness. Since previous studies showed that the memory performance for two depth planes did not significantly differ from that for one plane (Reeves & Lei, 2014; Qian et al., 2017), we only focused on the effect of depth order—that is, the difference between the front and the back planes within a two-plane layout. 

Experiment 1: Effect of relative saturation and brightness on VWM

In Experiment 1, we attempted to investigate the effect of relative saturation and brightness by dividing the memory items into two sets that differed both in saturation and brightness (S & B). Lowering S & B of the items reduces visual salience and may indicate a greater distance from the observer compared to items with high S & B, since aerial perspective phenomenon predicts that both the chromatic and achromatic contrasts of an object decrease as it recedes farther in distance. Furthermore, the joint effects of S & B and binocular disparity were tested by presenting the two sets of items at two disparity-defined depth planes. The disparity cue was manipulated to be either congruent or incongruent with the direction of depth implied by the aerial perspective cue (relative S & B) to examine how the two cues cooperate with or rival against each other. 

Method

Participants 

Fifteen students from Sun Yat-Sen University (SYSU) with normal or corrected-to-normal vision took part in the experiment for pay. All of them were naive to the purpose of the study. This research (Experiments 1–3) was approved by the SYSU Institutional Review Board. Written informed consent was signed by each participant prior to all the experiments. 

Apparatus and stimuli 

The stimuli were viewed against a uniform gray background (102.2 cd/m²) through a self-built Wheatstone stereoscope on a pair of 21-in. ViewSonics monitors. The display resolution was set to 1,600 × 900 pixels, with a refresh rate of 60 Hz. For the typical viewing distance of 70 cm, a pixel subtended approximately 1 arcmin. 

The memory array was composed of a set of colored squares, with their colors randomly selected without replacement from seven highly discriminable color categories: red, green, blue, yellow, magenta, cyan, and orange. The set size of the colored squares was four or six. The memory items were presented at pseudorandom positions within a 2 × 3 grid subtending 8° × 12°. They were randomly distributed among the six cells, with an angular separation of no less than 1° between any two adjacent items. Three experimental conditions of disparity manipulation were tested in separate blocks. In the none condition, no disparity was involved and all the items were presented at the plane of the monitor screen with half of high saturation and high brightness (HSHB; see Table 1 for HSV values, luminance values, and Michelson contrast: [L_max – L_bg] / [L_max + L_bg]), and the other half of low saturation and low brightness (LSLB; see Table 1 and Figure 1). In the congruent condition, the disparity cue was added and indicated the same direction of depth as relative S & B; in the incongruent condition, the disparity cue indicated the opposite direction of depth. In other words, HSHB items were at the stereoscopic front plane and LSLB items were at the stereoscopic back plane in the congruent condition, and vice versa in the incongruent condition. The two depth planes perpendicular to the line of sight were separated by a relative disparity of 0.2° (see Figure 1 for demonstration). An equal number of items were presented at the two depth planes. Each item subtended 0.65° × 0.65° of visual angle. However, due to the size constancy phenomenon, the items at the back plane appeared to be larger than those at the front. Therefore, we decreased the size of the items at the back so that their perceived sizes were matched to be the same. 

Table 1

 

Colors used in the study (HSV [hue, saturation, value]; luminance [cd/m²]; Michelson contrast). Notes: HSHB = high saturation high brightness; LSLB = low saturation low brightness; HSLB = high saturation low brightness.

 

Figure 1

 

Stimuli and procedure. Top left: Colors (red as an example) used in the study. Top right: The side view of the display in the congruent and incongruent conditions. Bottom: Task sequence for the congruent condition. For demonstration, the frames with solid lines indicate the front plane; the frames with dashed lines indicate the back plane. The frame lines were all solid in the formal experiments. HSHB = high saturation high brightness; LSLB = low saturation low brightness; LSHB = low saturation high brightness; HSLB = high saturation low brightness.

During the test phase, a probe was shown at one of the item locations (test location) in the memory array. Its color could either be the same as the memory item at the test location, or randomly varied to a new color category but with a HSHB or LSLB profile consistent with the memory item (e.g., memory item: HSHB red, probe: HSHB green). 

Procedure 

Observers were seated in a dark room to complete the whole experiment. Before the formal experiment, they needed to pass a screening test in order to make sure that they could perceive the disparity-defined depth. In the screening test, two horizontally displaced blue squares separated by a disparity of 0.2° were presented until response, and the observers were instructed to judge which one was nearer. They were required to achieve an accuracy of above 90% for 20 trials in order to continue with the formal experiment. They were then trained for a short time (2–5 min) to get acquainted with the stimuli and the task of the formal experiment. Each trial began with a fixation cross that subtended 0.3° × 0.3° presented at the center of the screen for 200 ms. In order for the observers to obtain stable depth perception of the stimuli, we first presented an array of squared frames at the locations of the future memory items for 2,000 ms. The memory array composed of colored squares was then presented for 250 ms. It was followed by a 900-ms blank retention interval and then the test phase. The probe remained on the screen until the observers responded. After the response, a 1,000-ms blank intertrial interval was presented before the next trial. A diagram of task sequence is shown in Figure 1. The observers were asked to judge whether the color of the probe remained the same as the memory item at the test location. If the same, they pressed the left arrow on the keyboard; otherwise, the right arrow. On 50% of the trials, the color of the probe changed. Each observer received 120 trials in which the probe was of HSHB and another 120 trials in which the probe was of LSLB, yielding a total of 240 trials. The order of the trials was randomized during the experiment. 

Results

The results are shown in Figure 2. We performed a 3 × 2 × 2 (Disparity Manipulation × Set Size × S & B) repeated-measures analysis of variance (ANOVA) to investigate the effect of relative S & B. The main effect of set size (four vs. six) was significant, F(1, 14) = 47.79, p < 0.001, Display Formula\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\(\eta _p^2\) = 0.77. The accuracy was higher for a set size of four than that for a set size of six. The main effect of S & B (high vs. low) was also significant, F(1, 14) = 31.83, p < 0.001, Display Formula\(\eta _p^2\) = 0.69, showing that the VWM performance for the HSHB items was better than for the LSLB items. The main effect of disparity manipulation was not significant, F(2, 28) = 0.83, p = 0.45, Display Formula\(\eta _p^2\) = 0.05, showing that applying disparity does not affect the overall performance. No significant interaction effect was found in the analyses (ps > 0.09). This suggests that contrasting items with relative S & B results in significant differences in the memory performance regardless of their locations in disparity-defined depth, supporting a visual saliency account. 

Figure 2

 

Results of Experiment 1. Left panel: The none condition. Middle panel: The congruent condition. Right panel: The incongruent condition. Error bars indicate standard errors here and in other figures.

Experiment 2: Effect of saturation on VWM

Since both the relative saturation and brightness of an item were varied in Experiment 1, the individual effects of relative saturation and relative brightness were still unclear. In Experiment 2, we manipulated the relative saturation, alone or in combination with disparity, to study their effects on VWM. The items in the memory array were divided into two sets that differed in relative saturation and displayed at two disparity-defined depth planes. 

Method

Participants 

Fifteen students from SYSU with normal or corrected-to-normal vision took part in the experiment for pay. All of them were naive to the purpose of the study. 

Stimuli and procedure 

The experimental procedure was identical to Experiment 1, except the following changes. In the none condition, all the items were presented at the plane of the monitor screen with half of HSHB and the other half of low saturation and high brightness (LSHB; see Table 1 and Figure 1). Note that theoretically, we could manipulate the saturation of a color while keeping its hue and physical luminance constant. However, perceptually, the hue would inevitably appear different, and it was not desirable to compromise the appearance of a color's hue in this experiment. In order to maintain consistent and reliable perception of hues between the HSHB and LSHB conditions, here we allowed small variations in luminance values. Since a previous study shows that the effect of saturation on depth perception seems to be modulated by contrast polarity (Dresp-Langley & Reeves, 2014), here the luminance contrast polarity with respect to the background was kept constant between the HSHB and LSHB sets (see Michelson contrasts shown in Table 1). In other words, if an HSHB item appeared to be brighter than the background, its LSHB counterpart also appeared to be brighter. In the congruent condition, HSHB items were at the front plane and LSHB items were at the back plane and vice versa in the incongruent condition. As in Experiment 1, the observers were asked to identify whether the color of the probe had changed compared to the memory item. Each observer received a total of 240 trials, with 120 trials in which the probes were of HSHB and 120 trials with probes of LSHB. 

Results

The results are shown in Figure 3. For data analysis, we performed a 3 × 2 × 2 (Disparity Manipulation × Set Size × Saturation) repeated-measures ANOVA. The main effect of set size (four vs. six) was significant, F(1, 14) = 126.99, p < 0.001, Display Formula\(\eta _p^2\) = 0.90. The main effect of saturation (high vs. low) was not significant, F(1, 14) = 2.47, p = 0.059, Display Formula\(\eta _p^2\) = 0.28, showing that the VWM performance for the HSHB items and for the LSHB items was not significantly different. The main effect of disparity manipulation was not significant, F(2, 28) = 0.31, p = 0.74, Display Formula\(\eta _p^2\) = 0.02. An interaction effect of Disparity Manipulation × Saturation was significant, F(2, 28) = 3.42, p = 0.046, Display Formula\(\eta _p^2\) = 0.32. Simple effect analysis showed that there was a significant difference between performance for HSHB items and LSHB items in the congruent condition (p = 0.01), but not in the incongruent condition (p = 0.11) nor in the none condition (p = 0.77). No other significant interaction effect was found in the analyses (ps > 0.11). This suggests that memory performance is not improved for items with high saturation unless they are located at a disparity-defined closer plane, supporting a depth perception account. 

Figure 3

 

Results of Experiment 2. Left panel: The none condition. Middle panel: The congruent condition. Right panel: The incongruent condition.

Experiment 3: Effect of brightness on VWM

Experiment 2 demonstrated a significant interaction effect of saturation and disparity manipulation on VWM, suggesting contributions of depth perception to affecting the performance. In Experiment 3, we further investigated whether combining the relative brightness and disparity cues could affect VWM. The items in the memory array were divided into two sets that differed in relative brightness and displayed at two disparity-defined depth planes. The items in the low brightness set were all darker than the background. 

Method

Participants 

Fifteen students from SYSU with normal or corrected-to-normal vision took part in the experiment for pay. All of them were naive to the purpose of the study. 

Stimuli and procedure 

The experimental procedure was identical to Experiment 1, except the following changes. In the none condition, all the items were presented at the plane of the monitor screen with half of HSHB and half of high saturation low brightness (HSLB; see Table 1 and Figure 1). In the congruent condition, HSHB items were at the front plane and HSLB items were at the back plane and vice versa in the incongruent condition. Note that decreasing the brightness of the items made them appear darker and inevitably less saturated, so the effect of brightness tested here might also include a slight contribution of the saturation effect. The task remained the same as in Experiment 1. 

Results

The results are shown in Figure 4. For data analysis, we performed a 3 × 2 × 2 (Disparity Manipulation × Set Size × Brightness) repeated-measures ANOVA. The main effect of set size (four vs. six) was significant, F(1, 14) = 48.33, p < 0.001, Display Formula\(\eta _p^2\) = 0.77. The main effect of brightness (high vs. low) was also significant, F(1, 14) = 9.62, p = 0.008, Display Formula\(\eta _p^2\) = 0.41, showing that the VWM performance for the HSHB items is better than for the HSLB items. The main effect of disparity manipulation was not significant, F(2, 28) = 0.32, p = 0.72, Display Formula\(\eta _p^2\) = 0.02. No significant interaction effect was found in the analyses (ps > 0.46). This suggests that contrasting items with relative brightness affects the memory performance regardless of their locations in depth, supporting a visual saliency account. 

Figure 4

 

Results of Experiment 3. Left panel: The none condition. Middle panel: The congruent condition. Right panel: The incongruent condition.

Control experiment: Rating depth separation

We ran a control experiment to test whether relative saturation and/or brightness could indeed serve as an aerial perspective cue and therefore could change the perceived depth, especially when combined with a disparity cue. The observers were instructed to rate the degree to which the two depth planes separated. 

Method

Participants 

Ten observers that passed the screening test described in Experiment 1 participated in the experiment for pay. All of them were naive to the purpose of the study. 

Stimuli and procedure 

Observers were seated in a dark room to complete the whole experiment. In each trial, two horizontally displaced colored squares were presented for 1,050 ms. The two squares were of the same hue, with one of HSHB and the other of low saturation and/or brightness (LSLB, LSHB, or HSLB in 1/3 of trials). Each pair of colors in the seven color categories was tested for 30 trials, with 10 trials for the none condition, 10 for the congruent condition, and 10 for the incongruent condition. A total of 630 trials were tested. The observers were instructed to rate the degree to which the two squares separated in depth. The ratings were made using a 5-point scale, on which 0 = no depth separation, ±1 = relatively small depth separation, and ±2 = a relatively large depth separation; a positive sign indicated that the HSHB item was perceived to be nearer, and a negative sign indicated that the HSHB item was perceived to be farther. Note that in the incongruent condition, a negative sign also indicated that the perceived direction of depth was consistent with that indicated by disparity. Observers were given 15 practice trials to get acquainted with the stimuli and the task. 

Results

The results of depth rating are shown in Table 2. For data analysis, a one-sample t test was performed for each condition to test whether the perceived depth separation was significantly different from 0 (no perceived depth). The p values of the t tests are shown in Table 2. The results showed that contrasting items solely with relative saturation created significant depth perception, while contrasting items with relative S & B or relative brightness created little depth perception (the none condition). In addition, stable depth perception could be observed in the congruent and incongruent conditions (ps < 0.007), except for the saturation manipulation in the incongruent condition (p = 0.094). The perceived direction of depth separation was consistent with that indicated by disparity in both the congruent and incongruent conditions, suggesting that disparity is a stronger depth cue than relative saturation/brightness. In general, the magnitude of the perceived depth separation was enhanced in the congruent condition (an average rating of 1.10) compared to that in the incongruent condition (an average rating of 0.83), especially when relative saturation was manipulated. This suggests that relative saturation and brightness could serve as a monocular depth cue—aerial perspective—to work in concert with the disparity cue. 

Table 2

 

Mean ratings ± standard errors (p values) of the control experiment. Notes: S & B = saturation and brightness; HSHB = high saturation high brightness; LSLB = low saturation low brightness; HSLB = high saturation low brightness.

 

Discussion

The present study investigated the effect of saturation and brightness on VWM by employing an adapted change detection paradigm. Manipulating saturation and brightness of the items may influence VWM performance through two potential mechanisms: visual salience and depth perception. The results of Experiments 1 and 3 showed that the overall memory performance for HSHB items was better than for LSLB and HSLB items, suggesting that visual salience affects the performance. Experiment 2 showed that the memory performance was not enhanced for HSHB items in the no-disparity condition or for HSHB items that were perceptually farther (the incongruent condition), but was only improved for HSHB items that were perceptually closer (the congruent condition), suggesting that depth perception also plays a role in affecting the memory performance. 

The results of Experiments 1 and 3 found that contrasting items with relative S & B or relative brightness could result in significant differences in the memory performance, regardless of their locations in disparity-defined depth. Items with high S & B are perceptually more salient, and research suggests that visual salience could benefit various perceptual performance by improving the bottom-up distinctiveness of a target or a location and consequently biasing attention to prioritize its processing (Fecteau & Munoz, 2006; Itti & Koch, 2000, 2001; Koch & Ullman, 1985; Treue, 2003). However, studies that investigated the effect of visual saliency on working memory arrived at contradictory conclusions. For example, Fine and Minnery (2009) found that visual salience of a target is positively correlated with one's ability to recall its spatial location, while Berg and Itti (2008) showed no significant correlation between the computed saliency of object patches and the recall rates in a memory task. In our study, if VWM is affected by visual salience, we would expect to observe a greater effect on memory performance with higher salience. Indeed, the effect size of S & B manipulation in Experiment 1 (Display Formula\(\eta _p^2\) = 0.69) was greater than the effect size of brightness manipulation in Experiment 3 (Display Formula\(\eta _p^2\) = 0.41). The effect of saturation in Experiment 2 was not significant, which is consistent with Olson et al.'s (2002) findings (Experiment 2). It is possible that variation in relative saturation alone is too trivial to produce a sufficiently high difference in salience; therefore, the memory performance cannot be affected. In addition to the contribution of visual salience, the worse performance for items with low brightness might be due to that changes between these items were harder to detect, which further resulted in a more difficult comparison process between the memory item and the probe. In our study, items with low brightness (LSLB and HSLB) were generally darker and had a smaller range of luminance variation compared to items with high brightness (HSHB and LSHB; see Table 1); therefore, they appeared to be more similar across different color categories. Research suggests that stimuli with higher similarity could result in a more difficult and erroneous comparison process (Awh et al., 2007; Jiang, Shim, & Makovski, 2008; Zhang & Luck, 2008). Therefore, in general, items with high brightness outperformed those with low brightness. 

Although visual salience may account for the effect of S & B and brightness in Experiments 1 and 3, it cannot explain the result pattern found in Experiment 2—VWM was enhanced for HSHB items at the front plane (the congruent condition), but not for HSHB items at the back plane (the incongruent condition). The memory items were identical in these two conditions but only the direction of the perceived depth differed, indicating the underlying mechanisms involved in this effect are different from that in the saliency effect, since color saliency is depth irrelevant and would not result in memory preference specifically for perceptually closer items. Because our control experiment shows that the magnitude of the perceived depth separation between the stereoscopically viewed closer items and farther items was greater in the congruent condition (1.17 ± 0.26) than in the incongruent condition (0.61 ± 0.32), the result pattern of Experiment 2 may reflect an effect of depth perception on VWM. Manipulating relative saturation could produce stable depth perception (p = 0.047; control experiment), and it works as an aerial perspective cue to cooperate with a congruent disparity cue to achieve a stronger and more stable depth perception (p = 0.001; control experiment) and further facilitated the memory performance for perceptually closer items. It may also rival with an incongruent disparity cue to deteriorate depth perception (p = 0.094; control experiment) and further diminished the benefit for memorizing closer items. The results are consistent with our previous study, where we found a memory advantage for perceptually closer items with depth perception induced by the congruent relative size and disparity cues (Qian et al., 2017). Therefore, we conclude that disparity-defined depth does play a role in influencing the VWM performance. Even though stereoscopic depth does not result in significant difference in memory performance by itself (Qian et al., 2017), it modulates the effect of saturation on VWM. 

In addition, we cannot completely rule out the possibility of an involvement of depth effect in Experiments 1 and 3. Although the interaction effect between S & B or brightness and disparity manipulations was not significant in Experiments 1 and 3, there was a consistent trend that the difference in change detection accuracy between the HSHB items and the LSLB/HSLB items in the congruent condition (an average of 9.5% in Experiment 1 and 6.6% in Experiment 3) was greater than that in the incongruent condition (an average of 6.5% in Experiment 1 and 3.7% in Experiment 3). Therefore, we think that the memory advantage for perceptually closer objects is still in effect, but this advantage is relatively small in these two experiments compared with the memory advantage produced by salience. The lack of significance in interaction may be due to that the saliency effect dominates over the depth effect in these two experiments. Combined with Experiment 2, which shows a significant interaction effect between saturation and disparity manipulations and an approach-significance saturation effect, the results of the three experiments suggest that both visual salience and depth perception affect VWM, but the question of “who is in charge” may depend on the degree to which an item is perceptually more salient or closer-to-observer than the others. 

Research on VWM suggests that both visual salience (Santangelo, 2015; Santangelo & Macaluso, 2013) and depth perception (Qian et al., 2017) improve memory performance through prioritizing certain items for attentional selection. As for the former, previous studies showed that bottom-up attention improves working memory performance via the order of encoding (Ravizza, Uitvlugt, & Hazeltine, 2016)—a hypothesis termed as prior entry (Santangelo & Macaluso, 2013). In other words, salient objects tend to be selected for encoding prior to other less salient objects and are more likely to reduce available resources to encode and then retrieve other objects (Pedale & Santangelo, 2015; Pooresmaeili, Bach, & Dolan, 2013). Therefore, memory performance for less salient items was decreased as some items became increasingly salient (Dempere-Marco, Melcher, & Deco, 2012; Melcher & Piazza, 2011). The effect of salience on VWM seems to be similar to the effect of depth perception, since our previous findings also show that the memory performance is enhanced for perceptually closer items and degraded for perceptually farther items within a two-plane layout, suggesting competition over resources (Qian et al., 2017). Because it is difficult to spread one's attention simultaneously over a 3-D space (Xu & Chun, 2007), one mostly attends to objects in depth sequentially (He & Nakayama, 1995). Ecologically, nearby objects may provide more relevant and usable information; therefore, we tend to allocate more attention to nearby objects at first and then switch attention to those far from us (Qian et al., 2017). As a result, nearby objects receive more cognitive resources for encoding, and are remembered better. In our study, combining the consistent depth cues makes depth perception more natural and realistic, and the enhanced depth perception consequently facilitates the process of “prior entry.” However, conflicting depth cues rival with each other, which makes depth perception less reliable, and little effect on VWM is observed. Taken together, we suggest that the effects of visual salience and depth perception on VWM are grounded in a common mechanism of attentional priority, which determines the order for encoding and the available resources for memorizing. 

One alternative explanation for the beneficial effect of depth perception on VWM is through a possible mechanism of grouping by depth (Reeves & Lei, 2014). Presumably, the visual system associates a depth tag with an object to indicate its egocentric distance (Qian, Liu, & Lei, 2016; Reeves & Lei, 2017), and objects with the same depth tag can be perceptually grouped to facilitate visual processing. According to this hypothesis, we would expect more objects to be retained in VWM across multiple depth planes than within a single depth plane. However, our previous study showed that the memory performance for a two-depth-plane layout does not significantly differ from that for a one-plane layout (Qian et al., 2017), suggesting that the enhancement observed in VWM is not due to grouping by depth. Overall, averaging the memory performance across different depth planes balanced out the memory advantage for perceptually closer items and the disadvantage for farther items. This is consistent with the evidence that simultaneously presenting the items in multiple depth planes does not improve memory performance (Reeves & Lei, 2014; Xu & Nakayama, 2007). Therefore, we rule out the grouping by depth hypothesis, and attribute the effect of depth to a mechanism involving in setting priority for attentional selection. 

In the present study, we used binocular disparity to induce depth perception, and manipulated relative saturation and brightness to provide a congruent or incongruent aerial perspective cue. Compared to binocular disparity, fewer studies have investigated the effect of relative saturation and brightness on depth perception. Egusa (1977) was the first study to show that the perceived depth separation between objects increased with increasing difference in brightness. Subsequently, Egusa (1983) found that the perceived depth separation also increased with increasing difference in saturation but the perceived direction of depth depended on hue, suggesting that depth perception is jointly determined by brightness, hue, and saturation. However, other studies showed that the depth effects produced by colors could be explained by variations in their luminance contrast irrespective of hue (Dresp-Langley & Reeves, 2012), and saturation systematically determined the perception of depth order—that is, strongly saturated regions appeared to be closer than weakly saturated regions (Dresp-Langley & Reeves, 2014). Our control experiment of depth rating showed that contrasting items solely with relative saturation created stable depth perception, while contrasting items with relative S & B or brightness created little depth perception. And generally, the difference in the perceived depth separation between the congruent and incongruent conditions indicates a contribution of relative saturation and brightness as a depth cue. 

Finally, our results show that depth perception may result from averaging or summing the effect of each depth cue. In our control experiment, the depth ratings in the congruent condition approximately equaled the sum of ratings in the none condition and the incongruent condition (absolute values), suggesting a summation effect of depth cues. Nevertheless, past research shows that depth perception can be determined by the direction of the dominating cue as well. Although some studies showed that monocular and binocular cues were combined by simple summation (Regan & Beverley, 1979), others suggested that cue summation occurred only when the two cues signaled the same directions of depth (Heuer, 1987). When cues indicated opposite directions of depth, they rivaled, leaving one dominating the perception (Howard, Fujii, & Allison, 2014; Qian & Petrov, 2012, 2013, 2016). Given these inconsistencies, the mechanism of depth cue interaction may be contingent on the types of depth cue employed and specific experimental settings, as suggested by Landy and Brenner (2001). 

Conclusions

The present study showed that the VWM performance was generally improved for items with higher saturation and higher brightness, but not for items solely with higher saturation. Combining the congruent disparity cue with relative saturation enhanced the memory performance for perceptually closer items; however, conflicting the two canceled this beneficial effect. These results indicate that both visual salience and depth perception affect VWM, possibly through a common underlying mechanism of setting priority for attentional selection. 

Acknowledgments

This work has been supported in part by the National Natural Science Foundation of China (31500919). 

Commercial relationships: none. 

Corresponding author: Jiehui Qian. 

Email: [email protected]. 

Address: Department of Psychology, Sun Yat-Sen University, Guangzhou, China.