Visual working memory and saliency independently influence the priority for access to visual awareness

Abstract

Both visual working memory (VWM) and visual saliency influence sensory processing, as is evident from research on visual attention and visual awareness. It is generally observed that items that are memorized or salient receive priority in visual search and in the access to awareness. Here we investigate whether these two factors interact and together boost access to visual awareness more than each factor independently. In the present experiment, we manipulated the VWM relevance and saliency of an item through a color memorization task and color uniqueness, respectively. We applied continuous flash suppression (CFS) to suppress items from visual awareness. The color of the suppressed items could either be congruent or incongruent with the memorized color, and either stood out from its surrounding distractors (salient pop out) or not. The item's priority for visual awareness was measured by measuring the time it took for an item to “break” into awareness. We first show that VWM relevance and visual saliency each shortened the time needed for an item to access awareness. More interestingly, the combined effect of VWM and visual salience was additive; that is, items that were both congruent and salient broke into visual awareness even faster. A race model further suggests that the interaction between these two mechanisms can be explained by statistical facilitation. Thus, VWM and saliency influence the priority to access visual awareness independently.

Introduction

Most information presented to our retina does not enter our visual awareness. Previous research has revealed that certain visual information is prioritized in entering visual awareness. For instance, upright faces, upright body, and recognizable words are known to receive priority to enter visual awareness compared to other information (Costello, Jiang, Baartman, McGlennen, & He, 2009; Jiang, Costello, & He, 2007; Stein, Senju, Peelen, & Sterzer, 2011; Yang & Yeh, 2011). One distinction that can be made between factors that determine the speed with which information enters visual awareness is that between bottom-up and top-down factors. 

First, visual awareness could be guided by bottom-up factors. For example, a green tomato stands out when surrounded by red tomatoes, resulting in higher saliency (Itti & Koch, 2000). To explore the influence of saliency on visual awareness, researchers have used interocular rivalry tasks by presenting different images to subject's eyes and investigate the processing of stimuli accessing visual awareness by measuring the perceptual dominance switch between the two eyes (Lin & He, 2009). Paffen, Naber, and Verstraten (2008) and Stuit, Verstraten, and Paffen (2010) varied item saliency and observed that salient items break interocular suppression faster than less salient ones. Furthermore, by using the recently developed interocular suppression paradigm called breaking continuous flash suppression (b-CFS; e.g., Gayet, Paffen, & Van der Stigchel, 2013; Jiang et al., 2007; Wang, Weng, & He, 2012), Gayet, Paffen, Belopolsky, Theeuwes, and Van der Stigchel (2016, experiment 2) replicated this effect. They showed that an increase in saliency through luminance contrast shortened the time to break interocular suppression, which resulted in subjects becoming aware of the originally suppressed stimuli faster. The b-CFS paradigm provides a controllable method to compare the potency of different visual stimuli to reach visual awareness. In a b-CFS paradigm, the detection of stimuli presented to one eye is initially suppressed from awareness by the continuous flash presented to the other eye (Tsuchiya & Koch, 2005). The time it takes for a suppressed stimulus to overcome the interocular suppression provides a measure of the priority a stimulus receives in entering to awareness (Gayet, Van der Stigchel, & Paffen, 2014). 

The influence of top-down factors on visual awareness has been studied by examining the influence of the visual working memory (VWM) content in b-CFS tasks. For instance, combining a VWM task with a b-CFS task, researchers investigated whether an item that was held in mind also entered awareness faster. The results suggest that items that matched VWM content got released from interocular suppression faster than the items that mismatched VWM content (Gayet et al. 2013; Liu, Wang, Wang, & Jiang, 2016; Pan, Lin, Zhao, & Soto, 2014; van Moorselaar et al., 2017). These studies suggest that VWM can affect visual awareness processing. 

Previous literature has shown that saliency and VWM can interact behaviorally. For example, by manipulating the saliency of items in a visual scene, researchers have observed that items with higher saliency improve subjects' ability to recall these items (locations; Fine & Minnery, 2009; Melcher & Piazza, 2011; Santangelo & Macaluso, 2013). In addition, VWM congruent items capture more attention than incongruent ones during a detection or search task (Downing 2004; Hollingworth, Matsukura, & Luck, 2013; Hollingworth, & Beck, 2016; Bahle, Beck, & Hollingworth, 2018; Olivers, 2009; van Moorselaar, Theeuwes, & Olivers, 2014; Soto, Heinke, Humphreys, & Blanco, 2005). Varying VWM validity and target saliency, Soto, Humphries, and Heinke (2006) reported that subjects detect a target faster when both salient and congruent with memory, as compared to when the target was either memory-congruent or salient. However, since the previous studies explored the interaction between saliency and VWM when the stimuli were not suppressed from awareness, it is currently unclear whether saliency and VWM can interact to act on the priority for entering visual awareness. In the current study, we aim to answer this question by combining a VWM task and a b-CFS task. This paradigm also allows us to investigate how visual working content and saliency interact in affecting priority to visual awareness. On each trial, subjects memorized one color for a later recall test and, between memorization and recall, detected the target location during the b-CFS period. During the CFS presentation, (a) we varied the target saliency by manipulating the pop-out of the suppressed stimuli by using a distinct color, and (b) we varied the target's match with memory content by changing whether the detected target was a memorized (match) or non-memorized (mismatch) color category. 

Given that both working memory congruence and higher saliency can affect priority for entering visual awareness, multiple outcomes are possible. When an item is salient as well as matching the content of VWM, reaction time (RT) in b-CFS could, in principle, be affected in three ways: subjects could become aware of the item (a) as fast as the salient or VWM-relevant item; (b) faster than either salient or VWM-relevant item but not faster than statistical facilitation; or (c) even faster than the statistical facilitation (the latter would resemble integration, as in multisensory processing). We will test a race model (Miller, 1982; Ulrich, Miller, & Shröter, 2007) to investigate whether integration of these two factors exists. The cumulative distributive functions (CDF) of RTs in the memory congruent condition and salient condition were used to calculate how fast responses in the memory congruent and salient condition are expected to be based on statistical facilitation (i.e., independent processing as indicated by the race model (RMI); see Miller, 1982; Ulrich, Miller, & Schröter, 1986). If responses to working memory congruent and salient conditions are faster than predicted by statistical facilitation, this indicates that integration between the visual working memory and saliency information exists. 

Method

Subjects

Twenty subjects participated in the current study for a monetary reward after signing informed consent. All the subjects were naïve to the research purpose. Their age ranged from 23 to 39 (M = 26.5, SD = 4.27; 13 men, 7 women). All subjects reported having normal or corrected-to-normal sight and having no visual disorder or epilepsy. 

Apparatus

The current experiment was conducted on a PC equipped with a linearized 27-in. LCD monitor (2560 × 1440 pixels, 144-Hz refresh rate) in a dark room. Stimulus presentation and response registration were controlled by MATLAB (R2016; MathWorks, Natick, MA) using the PsychToolbox extension (Brainard, 1997; Pelli, 1997). Subjects gazed at the monitor through a stereoscope with four mirrors (two per eye) to achieve dichoptic presentation for b-CFS. The viewing distance was maintained at about 61 cm with a chin and forehead rest. 

Task and conditions

The main part of a single trial consisted of a b-CFS task to test the effects of saliency and working-memory content on how fast targets break into visual awareness. As illustrated in Figure 1, all stimuli were presented on a uniform black background (7.00 cd/m²). To facilitate binocular fusion of the dichoptic images, the stimulus areas presented to each eye were enclosed by an identical Brownian (i.e., 1/f²) noise square frame with a height and width of 7° and a thickness of 0.4°. We presented a gray dot with a diameter of 0.27° at the center of each frame for fixation. The effect of working-memory content was manipulated by combining a color memory task with the b-CFS task (Figure 1A). 

Figure 1

 

(A) A schematic depiction of the sequence of events in a trial. Subjects were instructed to remember a color for later recall at the beginning of each trial and detect the location of target (e.g., the upright triangle) during the dynamic masks display. The target triangle color during the visual detection task could be working memory (in-)congruent as well as (non-)salient. (B) Examples of the different memory congruence and visual saliency conditions.

Before the b-CFS task, subjects were first shown a color for memorization (Figure 1A). This color could either match (congruent condition) or not match (incongruent condition) the color category of the target during b-CFS (Figure 1B). The effect of saliency was manipulated by moderating the stimuli's appearance. The suppressed stimuli could display three identical colors and one odd color (salient condition) or display four dissimilar colors (nonsalient condition). The memory congruency and saliency conditions could be combined in any way, resulting in a two-by-two factor design. 

Stimuli and procedure

Per trial, a color was randomly chosen from five main color categories (red, green, blue, brown, and purple) for memorization. Colors were equal in luminance to prevent that subjects relied on luminance instead of hue as a memorization strategy. To prevent a ceiling effect in task performance and a linguistic instead of visual memorization strategy, subjects had to choose the memorized one from two colors, of which one was the memorized color and the other one was a distractor color that looked similar but had a different hue (Olivers, Meijer, & Theeuwes, 2006; see Supplementary File S1 for details of generating the colors). In between the memorization probe and memorization test, subjects performed a b-CFS task (Figure 1A). The b-CFS method typically consists of the presentation of a target stimulus to the nondominant eye, and strong, dynamic masks to the dominant eye to ensure prolonged visual suppression of the target.¹ We measured each subject's dominant eye with a b-CFS task before the main experiment. We used b-CFS instead of another eye dominance test (e.g., hole-in-the-card) to determine because eye dominance is task specific and thus variable across tasks (Ding, Naber, Gayet, Van der Stigchel, & Paffen, 2018). We generated 200 different binary patterns (0 and 41.80 cd/m² for black and white parts, respectively) that consisted of pink noise images filtered by a Gaussian low-pass filter (ó = 3.2, Gayet et al., 2014), as the b-CFS masks. 

In the main task, subjects performed 20 practice trials, followed by 160 test trials. Every trial started by subjects pressing the space bar of the keyboard. After this, an upright colored triangle was presented at the center of fixation for 1000 ms (Figure 1A). Subjects were instructed to remember the color for later recall and were informed that the memory color was irrelevant for the suppression task. After a blank screen (1,250 ms), four colored triangles with randomly arranged different orientations (upright, upside down, leftward oriented, and rightward oriented) were presented to the nondominant eye at 2° center-to-center eccentricity on the cardinal axes (i.e., left, right, above, and below fixation dot). The intensity of the triangles increased gradually from 0% Michelson luminance contrast to a contrast of 26.35% within 1 s. We presented a different mask every 100 ms (10 Hz) in random order to the dominant eye. Subjects were required to keep fixating at the center dot and report the upright triangle (e.g., the target during CFS) location as soon as they located it. The target color could be either congruent or incongruent with the memory color in the memory-congruent or memory-incongruent conditions, respectively. The colors of the three nontarget triangles were chosen from the memory incongruent colors: the nontarget triangles shared a same color in the salient condition and had different colors in the nonsalient condition (see Figure 1 for specific examples). Considering the statistical power, we designed to balance the memory content congruent and incongruent trials to have the same number of trials (e.g., 50% for each). The suppression task lasted until a response was given or until 20 s without a response had passed. We presented text feedback in trials with an incorrect localization response. Trials with a localization error or without any response were recycled at the end of the experiment. Subjects received feedback about their accuracies in the working memory task every 10 trials. 

To exclude potential priming and predictive effects of the memory probe, we conducted a control experiment that was identical to the main experiment except for the fact that we removed the recall phase on each trial and instructed the subjects (n = 10) to just passively view the cue before the b-CFS task. 

Results

Subjects correctly indicated the b-CFS target location in 98.44% (SD = 2.00%) of the trials. Trials with incorrect localization during the suppression task were excluded from further analysis. The localization accuracy was analyzed using a two-factor repeated-measures ANOVA with the predictors memory congruence and saliency. We observed both main effects of visual saliency (salient vs. nonsalient: 98.94% [SD = 1.59%] vs. 97.93% [SD = 2.26%]; F(1, 19) = 5.89, p = 0.03) and memory congruence (congruent vs. incongruent: 98.93% [SD = 1.60%] vs. 97.94% [SD = 2.26%]; F(1, 19) = 6.48, p = 0.02), and a trend-level interaction, F(1, 19) = 3.27, p = 0.09. 

We compared the subjects' memory accuracies in memory-congruent and memory-incongruent conditions and observed no difference, m = 79.67%; t(19) = 1.12, p = 0.28. This suggests that the congruent color display during the suppression period did not affect subjects' performance in the memory test. 

The effects of congruence and saliency on RTs in the suppression task were also analyzed with a repeated-measures ANOVA. As illustrated in Figure 2, we observed significant main effects of memory congruence, F(1, 19) = 23.45, p < 0.001, and visual saliency, F(1, 19) = 18.39, p < 0.001, but no significant interaction between them, F(1, 19) = 1.79, p = 0.20. These results point at a faster breakthrough suppression by memory-congruent as compared to memory-incongruent targets (1440 ms vs. 1562 ms) and at visually salient targets having broken through suppression faster than nonsalient targets (1433 ms vs. 1569 ms). Furthermore, subjects detected targets that were both memory congruent and salient faster than targets that were only memory congruent or only salient (memory-congruent plus salient vs. memory-congruent plus nonsalient: 1323 ms vs. 1401 ms, t(19) = 3.69, p < 0.002; memory-congruent plus salient vs. memory-incongruent plus salient: 1323 ms vs. 1349 ms, t(19) = 3.71, p = 0.001). Considering the main effects of memory congruence and visual saliency on detection response accuracy reported above, the faster RTs in memory congruent and salient conditions rule out the possibility of speed–accuracy trade-off. 

Figure 2

 

Response times as a function of memory congruence and visual saliency for the main experiment. Error bars denote ±1 SEM.

The results of the control experiment show that the differences in RTs for the main experiment were not caused by priming or making use of the predictability of the cue, F(1, 9) = 0.03, p = 0.86: VWM-matching targets were not detected faster when passively viewing the cue. Interestingly, we even observed a negative priming effect; that is, subjects detected the target locations slower in the probed-plus-salient condition than in the salient-only condition, t(9) = 2.44, p < 0.04. These results indicate that the facilitation of detection times in the top-down condition cannot be explained by bottom-up priming or top-down prediction. 

In order to investigate whether the speedup in the memory congruent and salient condition compared to either memory congruent condition or salient condition could be explained by an independent processing model (i.e., working memory and saliency act independently on RT), or by a coactivation model (i.e., working memory and saliency integrate), we analyzed violations of a race model (Miller, 1982). The race model places an upper limit on the cumulative probability (CP) of a response at a given response time for redundant signals (i.e., the memory-congruent and salient condition in the current study). The race model holds, for any response latency, when the CP value from redundant signals does not exceed the sum of the CP from each of the single signals. If violated, this means that working memory and saliency information integrates. As illustrated in Figure 3, for the entire curve, the CP of observed memory congruent and salient condition was smaller than the sum of the CP of memory congruent condition and salient condition, indicating no violation of the race model. This suggests that working memory and saliency independently influence access to visual awareness. 

Figure 3

 

Redundancy gain analysis for violation of the race model inequality. The solid and dashed line stand for the cumulative probability distributions of RTs with memory-congruent and salient target and with the race-model–bound computed from memory-congruent target and salient target, respectively.

Discussion

The aim of the present study was to investigate (a) whether saliency and VWM-content interact in affecting priority for access to visual awareness, and if so, (b) how saliency and VWM interact. To this end, we combined a b-CFS task with a VWM task. We replicated two previously reported findings. First, we observed that VWM content-congruent items broke interocular suppression faster than incongruent items (Gayet et al., 2013; van Moorselaar et al., 2017). Second, we replicated that salient items receive priority over nonsalient items to break interocular suppression (Gayet et al., 2016; Paffen, Naber, & Verstraten, 2008; Stuit et al., 2010). Importantly, we observed that items that were both salient and memory-content congruent broke CFS even faster than either salient or memory-congruent items. This novel finding suggests that saliency information and VWM content interact in affecting priority for access to visual awareness. To study how saliency information and VWM interact, we conducted a race model analysis that revealed that the interaction between these two mechanisms can be explained by statistical facilitation. 

Saliency and VWM, as bottom-up and top-down factors, respectively, are known to act on sensory processing. Salient items are prioritized during search tasks (Theeuwes, 1991, 1992, 1994) and consciousness paradigms (Gayet et al., 2016; Naber, Carter, & Verstraten, 2009; Paffen et al., 2008; Stuit et al., 2010), even when subjects are not aware of the stimuli (Hsieh, Colas, & Kanwisher, 2011; Lin & Murray, 2015). For instance, McCormick (1997) originally observed that attention was captured by a subliminal exogenous cue, and a number of studies replicated this finding with different designs (for reviews, see Lamme & Roelfsema, 2000; Mulckhuyse & Theeuwes, 2010). Using a similar paradigm (e.g., CFS) as the current study, Hsieh et al. (2011) observed that a salient though invisible item captured more attention and subsequently improved sensory processing more than a less salient one. Similarly, VWM content-congruent items also receive priority in sensory processing, which results in an automatic capture of the focus of selective attention (Bahle, Beck, & Hollingworth, 2018; Hollingworth & Beck, 2016; Maxcey-Richard & Hollingworth, 2013; Olivers, 2009; Olivers et al., 2006; van Moorselaar, Theeuwes, & Olivers, 2014) and a boost in priority for access to awareness (Gayet et al., 2013; van Moorselaar et al., 2017). Our results are thus in line with these previous studies. 

The current study took the investigation into the role of bottom-up and top-down processes in prioritizing items one step further by combining their effects. In classical search paradigms (Theeuwes, 1991, 1992, 1994), subjects detect the target with a short reaction time, which could result in too limited temporal space to observe the possible additive effects of saliency and VWM (e.g., RTs cannot become faster due to a floor effect). The b-CFS paradigm used in the current study provides a proper method to avoid this issue. Typically, CFS delays the process of becoming aware of the items (e.g., seconds or even longer), allowing to measure the effects of saliency and working memory in a broader and therefore more sensitive range of RTs. This prolonged processing allowed us to observe the interaction between saliency and VWM directly in the current study. 

Our race model results reveal that salient and VWM content congruent items broke interocular suppression faster than either salient or VWM relevant items alone. However, the model did not violate statistical facilitation, suggesting that the visual system does not integrate (or “bind”) saliency and VWM-congruent information to multiply their effects. When an item is salient, processing is therefore not yet at its optimum; when the salient item also matches the content of VWM, processing can become even more efficient, resulting in even shorter RTs in the current experiment. The reverse also applies: When an item matches the content of VMW, processing can become even more efficient when the item is also salient. One other possible outcome option (integration) was also falsified: saliency and VWM information do not integrate in affecting visual processing; apparently, the visual system does not contain a mechanism akin that in multisensory processing; bottom-up saliency and top-down VWM information is not integrated in a specialized super-facilitation mechanism. Instead, the current results provide evidence that saliency and VWM content affect processing independently: On the one hand, items compete against each other to receive priority for visual processing based on the saliency (Itti & Koch, 2000). On the other hand, the VWM enhances the neural response to the concurrent visual input that matches the VWM content (Gayet et al., 2017). In conclusion, independent, parallel processing explains why we did not observe integration between saliency and VWM. 

To ensure that the difference in RTs between conditions reflected conscious access instead of a shift in post-conscious response criterion, previous studies included a monocular control experiment. Such an experiment includes the presentation of a single item (the target) and the dynamic pattern (e.g., masks in b-CFS condition) to the same eye and therewith the measurement of RTs without an episode of item invisibility due to visual suppression (Gayet et al., 2013; Jiang et al., 2007; Pan et al., 2014). No difference observed in RTs in the monocular task then (according to these studies) excludes the response criteria issue (Gayet et al., 2013; Jiang et al., 2007; Pan et al., 2014). However, van Moorselaar et al. (2017) observed a difference in RT between memory-congruent and memory-incongruent items in the monocular experiment, but only when an item was presented among distractors. This suggests that a part of the priority enhancement by VWM results from the interactions between items and distractors outside the operations of interocular suppression. Our aim was to study how saliency and VWM content interact in prioritizing access to awareness, thereby making the monocular control condition redundant. The results of the passively viewing control experiment suggest that it is unlikely that the subjects made use of the predictive value of the memory probe for the b-CFS task; however, it could still be argued that the items in the memory condition could be memorized more easily than the items in the passive-viewing condition. This limitation needs to be studied in further research. Although out of the scope of the current study, it would be interesting to investigate at what stages—from unconscious (e.g., by interocular and spatial suppression) through preconscious (e.g., only by monocular spatial suppression) to conscious—VWM and saliency operate. 

Acknowledgments

This project was supported by a China Scholarship Council (CSC) scholarship (201608330262) to author YD. 

Commercial relationships: none. 

Corresponding author: Yun Ding. 

Email: . 

Address: Department of Experimental Psychology, Utrecht University, Utrecht, the Netherlands. 

Footnotes

¹  To check whether the dynamic interocular masks suppressed the targets, we recruited 10 subjects and conducted the salient condition with and without interocular suppression. The subjects detected the target location significantly slower in the suppression condition than in the nonsuppression condition (difference = 438 ms, t(9) = 4.95, p < 0.001), which implies that our b-CFS manipulation was working as intended.