Free
Article  |   February 2013
Transfer of contextual cueing in full-icon display remapping
Author Affiliations
Journal of Vision February 2013, Vol.13, 2. doi:10.1167/13.3.2
  • Views
  • PDF
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to Subscribers Only
      Sign In or Create an Account ×
    • Get Citation

      Zhuanghua Shi, Xuelian Zang, Lina Jia, Thomas Geyer, Hermann J. Müller; Transfer of contextual cueing in full-icon display remapping. Journal of Vision 2013;13(3):2. doi: 10.1167/13.3.2.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract
Abstract
Abstract:

Abstract  Invariant spatial context can expedite visual search, an effect that is known as contextual cueing (e.g., Chun & Jiang, 1998). However, disrupting learned display configurations abolishes the effect. In current touch-based mobile devices, such as the iPad, icons are shuffled and remapped when the display mode is changed. However, such remapping also disrupts the spatial relationships between icons. This may hamper usability. In the present study, we examined the transfer of contextual cueing in four different methods of display remapping: position-order invariant, global rotation, local invariant, and central invariant. We used full-icon landscape mode for training and both landscape and portrait modes for testing, to check whether the cueing transfers to portrait mode. The results showed transfer of contextual cueing but only with the local invariant and the central invariant remapping methods. We take the results to mean that the predictability of target locations is a crucial factor for the transfer of contextual cueing and thus icon remapping design for mobile devices.

Introduction
Invariant visual context provides an important spatial cue for the guidance of visual search and focal-attentional selection. Repeated exposure to the same arrangements of search displays facilitates reaction time (RT) performance, an effect that has been referred to as contextual cueing (Chun, 2000; Chun & Jiang, 1998; Chun & Nakayama, 2000). In their seminal paper, Chun and Jiang (1998) had their observers search for a target letter “T” embedded in a set of distractor letters “L”. Unbeknown to participants, half of the presented displays contained identical configurations of target and distractor items (i.e., old displays), whereas the other half contained novel configurations (i.e., new displays). The main result was that of faster RTs to old relative to new displays (i.e., contextual cueing), an effect that developed after a short period of training. Interestingly, when observers were queried about repeated displays at the end of the search task in an “old-new” recognition test, their performance was only at chance level. From these findings, Chun and Jiang (1998) concluded that (a) contextual cueing guides focal attention more rapidly to the target location (but see Kunar, Flusberg, Horowitz, & Wolfe [2007], for evidence that contextual cueing might also aid postperceptual processes) and (b) the cueing effect derives from an implicit memory for the items' spatial arrangement. Since then, the cueing effect has been elaborated in a number of further studies (Chun, 2000; Chun & Jiang, 1998; Chun & Nakayama, 2000; Conci, Sun, & Müller, 2011; Conci & von Mühlenen, 2009, 2011; Geyer, Shi, & Müller, 2010; Jiang & Wagner, 2004; Kunar, Flusberg, & Wolfe, 2006). Jiang and Wagner (2004; see also Brady & Chun, 2007, or Olson & Chun, 2002) showed that contextual cueing is supported by two distinct spatial memory systems for individual item locations (i.e., local learning) and, respectively, the entire configuration formed by the distractors (i.e., global learning). Further, Kunar et al. (2006) showed that nonspatial attributes, too, such as background color, can facilitate RT performance. Contextual learning is also influenced by selective attention: Only the arrangement of some items, in particular, those sharing the target color, are learned over the course of an experiment (e.g., Geyer et al., 2010; Jiang & Leung, 2005). 
However, the degree to which contextual cueing can adapt to changes in learned displays remains subject to debate. For example, Jiang and Wagner (2004) reported that contextual cueing was still reliable even when learned displays were shifted along the horizontal display axis, the vertical display axis, or presented in a different size (compressed or expanded). Other studies (Brady & Chun, 2007; Olson & Chun, 2002) showed that contextual cueing survived changes of approximately 50% up to 75% of the display items; that is, cueing was reliable even when only one half or one quadrant of the display was repeated across trials. On the other hand, Olson and Chun (2002) reported that the cueing effect was abolished when new distractors were presented in between the target and the old distractors, with the target being presented, for example, in the left half and the old distractors in the right half of the display. Several other studies confirmed that contextual cueing diminished when the target was repositioned in repeated displays and thus became unpredictable (Chun & Jiang, 1998; Manginelli & Pollmann, 2009; Olson & Chun, 2002; Wolfe, Klempen, & Dahlen, 2000). In contrast, the contextual cueing effect remained effective with predictable target location changes (Conci & Müller, 2012; Conci et al., 2011). Makovski and Jiang (2010) suggested that predictability based on invariant context is a key factor for contextual cueing, based on their finding that the cueing effect decreased as the target appeared further away from its learned location; in fact, there were even RT costs when the target swapped its location with a previous distractor. Similar findings have been reported in three-dimensional (3D) scene search (Chua & Chun, 2003), in which contextual cueing decreased with increasing angular difference between viewpoints in the training versus the test displays (the experiment was divided into a training and test phase, with the latter containing modified displays). 
Although most of the work on contextual cueing was conducted using consistent (i.e., spatially invariant) search displays with a fixed number of items (e.g., one target and 11 distractors presented at a total of 48 locations within an invisible 6 × 8 matrix), none of these studies has examined the influence of changes of the display orientation on the cueing effect. Although changing display mode (and accordingly remapping of the items) occurs rarely with standard (i.e., laboratory) displays, switching display mode is a normal routine in current touch-based mobile devices, such as the iPad. Interestingly, with these devices, there is only one type of item—or icon—remapping method available: The positions of icons in one display (e.g., landscape mode) are remapped to the other display (portrait mode) by keeping the positional order (left to right and up to down) constant across all icons (see Figure 1a, b). Although this remapping method preserves the positional order and 80% of the horizontal intericon relationships (in a 4 × 6 icon matrix, as shown in Figure 1a, b), it destroys almost all local icon relationships, in particular, when the display is arranged as a rectangle (as with almost all mobile devices). However, based on the contextual cueing studies reviewed above, it is possible that contextual cueing is reduced, if not entirely abolished, when display orientation changes from landscape to portrait mode and icons are remapped in the standard position-order manner. Given this, one intriguing question arises, namely, are there any other improved methods for icon remapping, such that the remapping could enhance users' performance in everyday situations of display mode changes? This question was addressed in the current study by using the contextual cueing effect as a tool to evaluate the effectiveness of various display-remapping techniques; that is, preserved contextual cueing from one to the other display mode was taken as an indicator for the value of a given remapping method. 
Figure 1
 
Schematic illustration of display layouts and remapping methods. (A) Display layout in landscape mode; each number denotes an individual icon. (B) Portrait display layout obtained by the position-order invariant remapping method; the arrow indicates the icon remapping sequential order from the landscape to portrait mode. (C) Portrait display layout obtained by the global rotation remapping method; the arrow indicates the rotation direction from the landscape to portrait mode. (D) Portrait display layout obtained by the local invariant remapping method; circled regions remain the same between the landscape and portrait mode. (E) Portrait display layout obtained by the central invariant remapping method; circled regions are invariant.
Figure 1
 
Schematic illustration of display layouts and remapping methods. (A) Display layout in landscape mode; each number denotes an individual icon. (B) Portrait display layout obtained by the position-order invariant remapping method; the arrow indicates the icon remapping sequential order from the landscape to portrait mode. (C) Portrait display layout obtained by the global rotation remapping method; the arrow indicates the rotation direction from the landscape to portrait mode. (D) Portrait display layout obtained by the local invariant remapping method; circled regions remain the same between the landscape and portrait mode. (E) Portrait display layout obtained by the central invariant remapping method; circled regions are invariant.
Besides the position-order remapping method, several other (simple) remapping methods are possible. For example, one of the most natural ways is to rotate the entire display by 90° in the clockwise direction (individual icons are rotated 90° in the counterclockwise direction to keep their appearance constant; see Figure 1c). Such a global rotation is similar to the rotation of an object in our physical world (e.g., imagine you rotate a key cabinet with many keys). Alternatively, and motivated by the above mentioned studies on contextual cueing (e.g., Brady & Chun, 2007), one could also try to preserve local associations within the entire configuration as completely as possible. There are two ways to maximize such local invariants. One is to subdivide displays into several local regions and preserve the placement of these local regions in the entire configuration after icon remapping (Figure 1d). Another method is to keep the display center constant in remapped displays (Figure 1e). 
To investigate how these various display-remapping methods influence memory performance, we examined contextual cueing effects in four separate experiments. Each experiment examined one display remapping method. To simulate touch-based icon displays and observers' active touch action, we used real desktop icons as search items and presented them on a touch monitor in the four experiments. 
Methods
Participants
A total of 40 observers took part in the experiments (10 in each experiment, mean ages: 27.9, 26.2, 25.5, and 27.3 years and number of females: 7, 6, 6, and 5 for Experiments 1–4, respectively). All had normal or corrected-to-normal visual acuity (including color vision). They gave written consent prior to the experiment and were paid at a rate of €8/hour for taking part. Participants were naive as to the intention of the study. 
Apparatus and stimuli
The experiments were conducted in a dimly lit cabin (ambient light: 4.36 cd/m2). Visual stimuli were presented on a 23-inch multitouch LCD monitor (HP2310ti) with spatial resolution set to 1920 × 1080 pixels. To make touch pointing comfortable for the participants, the screen panel was placed on the table tilted by 45°. The viewing distance was approximately 40 cm, with participants' head position fixed by a chin rest. Twenty-four typical computer icons (randomly selected from 48 candidate icons1 for each observer) were presented within an invisible 6 × 4 horizontal grid (subtending 24° × 16° of visual angle) or a 4 × 6 vertical grid (subtending 24° × 16°). The target was the icon with a top overlay of a compound letter “T” (subtending 1.6° × 1.6°; luminance 35.67cd/m2; see Figure 2). Such a compound target letter was used for two reasons: first, to avoid interference between the target and some other (distractors) letters, and second, to make the compound letter and the icon comparable in terms of their luminance level. The background of the search displays was set to gray (16.56 cd/m2). To enhance the global spatial “Gestalt” (i.e., perception of the display as landscape or, respectively, portrait mode), we added one array of six upright white triangles (130.5 cd/m2) with a gray background (19.62 cd/m2) below the landscape mode (Figure 2a) or to the left side of the portrait mode (Figure 2b). The triangle array was meant to serve as a global landmark in the experiments, indicating display mode changes. The experimental program was developed with and controlled by Matlab (Mathworks Inc., Natick, MA), in addition to the Psychtoolbox (Brainard, 1997; Pelli, 1997). Response times were recorded via the touch screen. To determine the onset of a response, an additional input button (connected to a NI PXI system) was placed in between the touch screen and the participants, which was used for initiating the task and pointing movement. 
Figure 2
 
Example displays in the experiments. (A) Example of a landscape display. In this example, the “Apple” icon (second row, right-most column) is the search target. (B) Example of a portrait display. In this case, icons are remapped from the landscape mode by keeping the position order constant in the left-to-right and up-to-down manner (Experiment 1). (C) The top overlay for the target icon (a compound letter “T”).
Figure 2
 
Example displays in the experiments. (A) Example of a landscape display. In this example, the “Apple” icon (second row, right-most column) is the search target. (B) Example of a portrait display. In this case, icons are remapped from the landscape mode by keeping the position order constant in the left-to-right and up-to-down manner (Experiment 1). (C) The top overlay for the target icon (a compound letter “T”).
Design and procedure
A three-factorial within-subject design was used with display mode (landscape, portrait), context (old, new), and experimental epoch (1–9) as independent variables. From the 24 possible target locations, we randomly selected 12 target positions for old and the other 12 positions for new displays. In this way, the target appeared equally likely at any of the 24 possible locations. To have enough difference between old and new configurations and to control the similarity of icon identities, we selected 24 icons from 48 typical icon candidates and assigned to random locations. Each of the new target locations was paired with newly generated distractor icons for every new-display trial, whereas each of the old target locations was paired with randomly selected distractor icons at the beginning of each experiment and served as old landscape displays. These old landscape displays were also used to define the remapped old portrait displays. Remapping was one as follows: 
  •  
    Experiment 1 (position-order invariant). The positional order (left to right and top to bottom) of the icons in the portrait mode was the same as that in the landscape mode (Figure 1b). This method is used in most of the present mobile devices for the rearrangement of icons.
  •  
    Experiment 2 (global-rotation). The landscape display was, as a whole, rotated by 90° clockwise into the portrait mode, while preserving the (upright) orientation of the individual icons. With this global rotation, the global and local relationships of the icons are rotated by 90° across display changes (Figure 1c).
  •  
    Experiment 3 (local invariant). To preserve the local (and global) spatial configuration as much as possible, in Experiment 3, the display was divided into four peripheral and one central region, each consisting of four icons (see circled regions in Figure 1d). The positioning of these four “corners” and the central region were kept constant across display mode changes. Only four remaining items (i.e., Icons 3, 4, 21, and 22 in Figure 1d) changed their relative positions. Similar to the global rotation, with the local-invariant transformation, the local relationships between all icons are preserved across display changes.
  •  
    Experiment 4 (central-invariant). Instead of dividing the display into multiple regions, in Experiment 4, we preserved the central display region as much as possible (i.e., preserving the central maximum square region). As shown in Figure 1e, icons in the central 4 × 4 matrix were positioned at identical locations across display mode changes. In addition, the four outermost (corner) icons were also unchanged. Only the remaining four icons (7, 12, 13, and 18 in Figure 1e) changed their positions.
Each experiment comprised three consecutive sessions: learning, test, and recognition. In the learning session, there were five epochs of three blocks, with each block consisting of 24 search trials. To keep the experiment as short as possible, the learning session contained only 12 old-landscape displays to foster learning effect (each of the old display repeated twice per block). The transfer session had four epochs, with each epoch consisting of 24 trials (i.e., one block only). In half of these trials, an old display was presented and new displays in the other half. New displays were randomly generated at the beginning of each trial. The order of display modes in the transfer epochs was fixed: landscape (L), portrait (P), portrait (P), and landscape (L). The first transfer epoch with the landscape mode (i.e., nontransformed) was intended to test for a standard contextual cueing effect. The last transfer epoch was intended for examining whether contextual cueing is still manifested by two intervening epochs containing different display modes. To avoid confounding by repetition effects, we randomly presented trials in such a way that the same old display was never repeated within three consecutive trials. 
In the learning and test sessions, each trial started with a cross-fixation presented in the center of the display. Participants had to press the input button (also serving as the initial hand position) to trigger the presentation of the search display. Participants were instructed to detect the target and touch its location with their index finger as rapidly and as accurately as possible. A blank screen was presented after the localization response, or 4.5 s when no response was made. When participants made an erroneous response, an additional feedback display containing a stop warning sign was presented for 1.0 s. After 1.0 to 1.2 s of intertrial interval, the next trial started. 
In the recognition session, participants were asked if they had realized any display repetitions during the learning and transfer sessions and, if so, when they had first noticed the repeated displays (note that a similar protocol was used by Chun & Jiang, 1998). Following this, they had to judge a total of 24 displays, including 12 new displays (six landscape and six portrait displays) and 12 old displays (six landscape and six portrait displays), in an “old-new” recognition test. In this test, the chance rate for recognizing a repeated display was 50%. 
Prior to the experiment, participants practiced the experimental task in one training block of 24 trials (data not recorded). The search displays used in the practice trials were not shown later in the experiment. Participants were allowed to take a break in between successive blocks of the experiment. The break between the learning and transfer sessions was similar to other between-block breaks. 
Results
Accuracy performance
Error rates were overall small (<1%) and were comparable across all experiments. For further RT data analyses, we excluded trials with erroneous responses and RTs outside the range of 200 to 3000 ms. Such outliers were also low in general (<3%). 
Perceptual learning
The mean RTs for the learning sessions are shown in Figure 3 (Epochs 1–5). For each experiment, the mean RTs were examined by repeated-measures analysis of variance (ANOVA) with the single-factor epoch. The main effect was significant for all four experiments (all p's < 0.05); further Bonferroni tests revealed a significant perceptual learning effect, defined as the difference in RTs between Epoch 5 (i.e., the end of the training session) and Epoch 1 (i.e., the beginning of the training session; Table 1). In addition, to examine interference by the introduction of new (both landscape and portrait) displays in the transfer session, RTs for the old displays in the first epoch of the transfer session (Epoch 6) were compared with RTs in the last epoch of the learning session. Although RTs were numerically longer in Epoch 6 compared with Epoch 5, the slowing was significant only for Experiment 2 (Table 2). This suggests that introducing novel displays had only some moderate influence on the search task response. 
Figure 3
 
Mean correct response times (RTs) as a function of epoch for the learning (Epochs 1–5) and transfer (Epochs 6–9) sessions; for the latter, mean RTs are shown separately for old displays (denoted by diamonds and solid lines) and new displays (denoted by dots and dashed lines). (A) Experiment 1, position-order invariant remapping. (B) Experiment 2, global rotation remapping. (C) Experiment 3, local invariant remapping. (D) Experiment 4, central invariant remapping.
Figure 3
 
Mean correct response times (RTs) as a function of epoch for the learning (Epochs 1–5) and transfer (Epochs 6–9) sessions; for the latter, mean RTs are shown separately for old displays (denoted by diamonds and solid lines) and new displays (denoted by dots and dashed lines). (A) Experiment 1, position-order invariant remapping. (B) Experiment 2, global rotation remapping. (C) Experiment 3, local invariant remapping. (D) Experiment 4, central invariant remapping.
Table 1
 
Mean learning effect in the training sessions and interference by the addition of new displays in the transfer session, for each experiment.
Table 1
 
Mean learning effect in the training sessions and interference by the addition of new displays in the transfer session, for each experiment.
Perceptual learning Interference associated with the presentation of new displays
Experiment Facilitation (ms) ANOVA Cost (ms) ANOVA
1 364 p < 0.01 60 p = 0.08
2 185 p < 0.05 154 p < 0.05
3 385 p < 0.01 36 p = 0.21
4 380 p < 0.01 98 p = 0.11
Table 2
 
Contextual cueing effects in the transfer session. The reported significance values are as follows: *p < 0.05; **p < 0.01; ***p < 0.001.
Table 2
 
Contextual cueing effects in the transfer session. The reported significance values are as follows: *p < 0.05; **p < 0.01; ***p < 0.001.
Experiment Contextual cueing effect (ms) ANOVA test with F value
Average (Epoch 6–9) Epoch 6 Epoch 7 Epoch 8 Epoch 9 Context (old/new) Epoch (6–9) Interaction
1 86 202*** 5 18 120* 9.25* 9.23** 4.49*
2 120 262*** 7 28 197** 12.48** 1.40 8.80***
3 156 223** 50 147* 205** 21.49*** 5.16** 3.07*
4 163 286** 43 142* 190** 22.42*** 5.63** 3.91*
Transfer of contextual cueing effect
The mean RTs, separately for old and new contexts, as a function of epoch for the test phase are presented in Figure 3 (Epochs 6–9). To examine the contextual cueing effect, mean RTs were subjected to a repeated-measures ANOVA with epoch (6–9) and context (old vs. new) as factors, separately for each experiment. The results are summarized in Table 2. The RTs were significantly faster for old displays compared with new displays in all four experiments, indicating robust contextual cueing benefits. The main effect of epoch was also significant for Experiments 1, 3, and 4, indicating that some perceptual learning also occurred in the transfer session. Finally, the context × epoch interaction was significant for all experiments, reflecting differential cueing effects in the different epochs. Post hoc tests revealed significant contextual cueing to be significant for all landscape displays (Epochs 6 and 9). By contrast, for portrait displays (in Epoch 8), significant contextual cueing was evident only in Experiments 3 and 4 (see Table 2). Note that each epoch in the transfer session contained only 24 trials, suggesting that the contextual cueing effect could be quickly transferred with the local invariant and central invariant remapping methods when the display mode was changed. 
To examine whether contextual cueing effects were comparable among the different experiments, a repeated-measure ANOVA was conducted on the cueing effect in the first transfer Epoch 6 (with landscape mode), with the single-factor experiment. The effect of experiment was nonsignificant, F(3, 27) = 0.55, p = 0.65, suggesting that the contextual cueing effects were comparable among experiments. Thus, any differences in the subsequent transfer epochs are likely attributable to the particular method of display (icon) remapping. 
Recognition test
Based on participants' postexperimental reports, we determined the percentages of participants who noticed display repetitions during the search task and who attempted to explicitly learn the displays; the times (in terms of the number of blocks performed) at which these participants first noticed the repetitions were also calculated. We then further calculated participants' mean hit and false alarm rates as well as their discrimination sensitivities (d′) for landscape (L) and portrait (P) displays. The results are summarized in Table 3
Table 3
 
Results of recognition test. The reported significance values are as follows: *p < 0.05; **p < 0.01.
Table 3
 
Results of recognition test. The reported significance values are as follows: *p < 0.05; **p < 0.01.
Experiment Noticed repetition Explicit learning When (blocks) Hit rates False alarms d′L d′P d′L = d′P
1 90% 60% 6.89 76.3% 38.3% 2.17** 1.17* p = 0.17
2 80% 30% 7.25 72.9% 43.3% 1.33** 1.17* p = 0.68
3 90% 20% 5.85 65.1% 28.3% 1.56 2.02* p = 0.51
4 80% 60% 4.75 80% 28.3% 3.42** 1.52* p < 0.05
In all experiments, participants exhibited high proportions of recognized displays. The recognition sensitivities (d's) were significantly larger than zero for both landscape and portrait displays (p < 0.05), except for one marginally significant effect for the landscape display in Experiment 3 (p = 0.066), which was mainly due to one observer who showed an extreme negative dH′ score (−1.40). When excluding this participant, dH′ was also significant: p < 0.05. Taken together, the significantly positive d′ scores suggest that after learning, participants recognized not only the old landscape displays but also the remapped portrait displays in all four experiments. Moreover, there was no significant difference in recognition sensitivity between the landscape and portrait displays, at least for the first three experiments (see the last column in Table 1), indicating that remapping did not hamper explicit recognition. Although recognition accuracy was lower for portrait than for landscape displays in Experiment 4, the effect was mainly due to the very high recognition sensitivity in the landscape mode (Table 1). Nevertheless, even in Experiment 4, the sensitivity for the portrait displays was still significantly greater than zero, supporting the idea that the transformed old portrait displays can be recognized explicitly. The lack of differential recognition sensitivities between landscape and portrait displays in Experiments 1 and 2 is in contrast to the differential contextual cueing effects with landscape versus portrait displays. This suggests that recognition and visual search may involve different memory processes, with the former recruiting more complex information-matching processes that do not benefit the search processes. 
Discussion
The present study examined the transfer of learned contextual cues in full-icon display remapping. The main goal was to investigate whether contextual cueing continues to facilitate icon localization (RT) performance following display mode changes. We compared four different types of icon remapping: position-order invariant, global-rotation, local-invariant, and central-invariant remapping. In all experiments, robust learning effects were found in the training session for the landscape displays. The RTs were faster at the end relative to the beginning of the training session. This practice effect is likely attributable to general learning of the localization task (Schneider & Shiffrin, 1977). 
In the test session, in which new displays were introduced (in addition to the old displays), we established a contextual cueing effects in all experiments, at least when the display mode was kept the same. This suggests that icon identities and spatial configurations among icons could serve as context cues to facilitate the localization task. Note that the facilitation effect might also be partially due to position-based learning, given that only old displays were used in the training session. However, the transfer effects found in the portrait displays (Experiments 3 and 4) cannot be fully explained by position-based learning, because the positions were changed in the portrait displays and positional repetitions were equated between the old and new displays. Interestingly, contextual cueing was evident for landscape displays even after the insertion of two epochs of portrait displays. This may be taken to indicate that the cueing effect is relatively robust against interference within the same set of old configurations, consistent with previous studies (Chun & Jiang, 1998, 2003; Conci et al., 2011; Conci & Müller, 2012; Jiang, Song, & Rigas, 2005; Song & Jiang, 2005; Zellin, Conci, von Mühlenen, & Müller, 2011). However, contextual cues acquired with landscape displays were transferred to portrait displays only under certain remapping conditions (those of Experiments 3 and 4), suggesting that contextual cueing is relatively inflexible and that transfer is confined to specific remapping situations. 
The differential pattern of effects revealed among the four experiments raises the question as to the factors that modulate the transfer of learned displays. The position-order invariant method maintained icons in their same left-to-right and up-to-down manner. Although 80% of the horizontal relationships are preserved with this transformation, it destroys almost all vertical relationships. It also changes the absolute positions of the icons dramatically; for instance, Position 5 is shifted from the left side in the landscape display (Figure 1a) to the right side in the portrait display (Figure 1b). As a result, the target location might become unpredictable in remapped displays, abolishing the contextual cueing effect (Conci et al., 2011; Manginelli & Pollmann, 2009). Note that in the current terms, predictability refers to both the target's absolute location on the screen as well as its placement within the entire configuration (given that we did not vary the target's absolute and relative location independently). 
When comparing the position-order invariant to the global-rotation method, the latter maintains all local icon neighborhood relationships, but the overall Gestalt is rotated by 90° from the landscape to portrait mode. With this type of remapping, repeated displays failed to facilitate RT performance in portrait displays. Possibly, the contextual associations learned in the landscape displays were quite instance specific and too weak for the global-rotation remapping. As shown in mental rotation studies (Böckler, Knoblich, & Sebanz, 2011; Borst, Kievit, Thompson, & Kosslyn, 2011; Ionta & Blanke, 2009; Shepard & Metzler, 1971; Shomstein & Yantis, 2004), RTs increase linearly with increasing angular disparity when participants were asked to decide whether two presented objects are the same. Those paired objects were normally rotated objects or mirrored objects, and participants had to carry out mental rotation (rotating one object into the other) to solve the task. Applied to the current Experiment 2, although the global rotation maintains the local icons' neighborhood relationships, the mapping of a new portrait onto an old landscape display may likewise be a demanding (i.e., time-consuming) process, which diminishes any performance gains brought about by contextual cueing. In a previous study using 3D visual search, Chua and Chun (2003) also showed that contextual cueing decreased with increasing angular difference between viewpoints of training and test displays. Thus, demanding mental rotation might be the main reason why we failed to find any transfer of contextual cueing from the landscape to the portrait in Experiment 2. It should be noted, however, that in our setup, the experimental program presented the rotated portrait display automatically. That is, participants passively viewed the search displays, rather than carrying out the rotation actively. It would be interesting to examine the transfer of contextual cueing when participants rotate the displays themselves (i.e., actively). 
In contrast to Experiments 1 and 2, we found significant transfer of contextual cueing in Experiments 3 and 4, in which the portrait display was remapped from the landscape display using the local-invariant (Experiment 3) or central-invariant methods (Experiment 4). Both experiments disclosed numerical contextual cueing benefits already in the first epoch with portrait displays (50.8 and 44 ms for Experiments 3 and 4, respectively), although these effects were not significant. No contextual cueing in the first portrait epoch is likely due to the orientation change of the whole display. Mapping old landscape to portrait displays may engage additional mental processes, diminishing the contextual cueing effect. In addition, interobserver variability was large because both the old and new displays were presented only once in this epoch. Interestingly, transfer of contextual cueing was highly reliable for both remapping methods (147.4 and 142.0 ms for Experiments 3 and 4, respectively) in the second epoch. The local-invariant remapping method keeps five of seven local regions unchanged, and the global topological relationship of these five local regions also remains the same. This means that local regions appear at the very same positions (quadrants) in the entire configuration after the remapping. Likewise, the central-invariant remapping method maintains the absolute icon positions of the four outermost corners and the central region (83% in total). In both cases, after the remapping, the target position is much more predictable compared with both the position-order invariant and the global rotation methods. In contrast to previous investigations of contextual cueing, suggesting that only three to four repeated items (among some eight novel items) can produce the effect (Song & Jiang, 2005), the results of the present Experiments 1 and 2 suggest that merely preserving some local invariant information does not guarantee transfer of contextual cueing. Instead, the three to four items would have to appear at the very same positions within the global configuration to observe contextual cueing (Experiments 3 and 4; see also Brady & Chun, 2007, for a related proposal, albeit using different approach). 
The recognition tests showed that in all experiments, participants were well able to discern repeated from nonrepeated displays. This contrasts with standard contextual cueing studies in which recognition accuracy was typically at chance level (Chun & Jiang, 1998). Explicit memory effects may be due to the heterogeneous and, importantly, realistic icons used as distractors in our experiments (see also Brockmole, Castelhano, & Henderson, 2006). Interestingly, in all the experiments of the present study, recognition accuracy was larger than chance for all landscape and, importantly, remapped portrait displays. In contrast, transfer of contextual cueing was observed only in Experiments 3 and 4. This argues that merely recognizing a repeated display as an old one does not necessarily mean that this also facilitates RT performance. Of interest in this regard, it has been reported that explicit learning of repeated displays engages neural processes that are distinct from those concerned with implicit configural learning (Geyer, Baumgartner, Müller, & Pollmann, 2012; Preston & Gabrieli, 2008; Westerberg, Miller, Reber, Cohen, & Paller, 2011). Along these lines, we suggest that recognition and visual search are supported by different memory processes. Further, the dissociation between the transfer of contextual cueing (Experiments 3, 4) and explicit recognition (Experiments 1–4) suggests that the memory underlying explicit learning is more flexible than that underlying implicit configural learning. 
Conclusion
In sum, our experiments suggest that when display orientation switches and icons are rearranged, the traditional position-order remapping method used in current mobile touch devices is suboptimal in aiding search performance. Comparing and contrasting three alternative methods of icons remapping, we found that when using local-invariant or central-invariant remapping, contextual cueing continues to enhance (target) icon localization performance. Although the global-rotation method may be intuitive for users, it might introduce additional mental-rotation processes that are detrimental to localization performance. Our findings thus have implications for alternative interface design guidelines for icon rearrangement in mobile devices. Open questions awaiting further research concern how to optimize local invariance regions and what the effects of active manual rotation are. 
Acknowledgments
This work is supported in part by German Research council (DFG) project grant SH166 to Z. S. and by Chinese Scholarship Council (CSC) to X. Z. and L. J. 
Commercial relationships: none. 
Corresponding author: Zhuanghua Shi. 
Email: shi@psy.lmu.de 
Address: Experimental Psychology, Department Psychology, Ludwig-Maximilians-Universität München, 80802, Munich, Germany. 
References
Böckler A. Knoblich G. Sebanz N. (2011). Giving a helping hand: Effects of joint attention on mental rotation of body parts. Experimental Brain Research,211, 531–545. doi:10.1007/s00221-011-2625-z. [CrossRef] [PubMed]
Borst G. Kievit R. A. Thompson W. L. Kosslyn S. M. (2011). Mental rotation is not easily cognitively penetrable. Journal of Cognitive Psychology,23, 60–75. [CrossRef]
Brady T. F. Chun M. M. (2007). Spatial constraints on learning in visual search: Modeling contextual cuing. Journal of Experimental Psychology: Human Perception and Performance,33, 798–815. doi:10.1037/0096-1523.33.4.798. [CrossRef] [PubMed]
Brainard D. H. (1997). The psychophysics toolbox. Spatial Vision,10, 433–436. [CrossRef] [PubMed]
Brockmole J. R. Castelhano M. S. Henderson J. M. (2006). Contextual cueing in naturalistic scenes: Global and local contexts. Journal of Experimental Psychology: Learning, Memory, and Cognition,32, 699–706. doi:10.1037/0278-7393.32.4.699. [CrossRef] [PubMed]
Chua K. P. Chun M. M. (2003). Implicit scene learning is viewpoint dependent. Perception & Psychophysics,65, 72–80. [CrossRef] [PubMed]
Chun M. M. (2000). Contextual cueing of visual attention. Trends in Cognitive Sciences,4, 170–178. [CrossRef] [PubMed]
Chun M. M. Jiang Y. (1998). Contextual cueing: Implicit learning and memory of visual context guides spatial attention. Cognitive Psychology,36, 28–71. doi:10.1006/cogp.1998.0681. [CrossRef] [PubMed]
Chun M. M. Jiang Y. (2003). Implicit, long-term spatial contextual memory. Journal of Experimental Psychology: Learning, Memory, and Cognition,29, 224–234. [CrossRef] [PubMed]
Chun M. M. Nakayama K. (2000). On the functional role of implicit visual memory for the adaptive deployment of attention across scenes. Visual Cognition,7, 65–81. doi:10.1080/135062800394685. [CrossRef]
Conci M. Müller H. J. (2012). Contextual learning of multiple target locations in visual search. Visual Cognition,20, 746–770. doi:10.1080/13506285.2012.694376. [CrossRef]
Conci M. Sun L. Müller H. J. (2011). Contextual remapping in visual search after predictable target-location changes. Psychological Research,75, 279–289. [CrossRef] [PubMed]
Conci M. von Mühlenen A. (2009). Region segmentation and contextual cuing in visual search. [Research Support, Non-U.S. Gov't]. Attention, Perception & Psychophysics,71, 1514–1524. doi:10.3758/APP.71.7.1514. [CrossRef] [PubMed]
Conci M. von Mühlenen A. (2011). Limitations of perceptual segmentation on contextual cueing in visual search. Visual Cognition,19, 203–233. [CrossRef]
Geyer T. Baumgartner F. Müller H. J. Pollmann S. (2012). Medial temporal lobe-dependent repetition suppression and enhancement due to implicit versus explicit processing of individual repeated search displays. Frontiers in Human Neuroscience, 6(272), doi:10.3389/fnhum.2012.00272.
Geyer T. Shi Z. Müller H. J. (2010). Contextual cueing in multiconjunction visual search is dependent on color- and configuration-based intertrial contingencies. [Research Support, Non-U.S. Gov't]. Journal of Experimental Psychology: Human Perception and Performance,36, 515–532. doi:10.1037/a0017448. [CrossRef] [PubMed]
Ionta S. Blanke O. (2009). Differential influence of hands posture on mental rotation of hands and feet in left and right handers. Experimental Brain Research,195, 207–217. doi:10.1007/s00221-009-1770-0. [CrossRef] [PubMed]
Jiang Y. Leung A. W. (2005). Implicit learning of ignored visual context. Psychonomic Bulletin & Review,12, 100–106. [CrossRef] [PubMed]
Jiang Y. Song J. H. Rigas A. (2005). High-capacity spatial contextual memory. Psychonomic Bulletin & Review,12, 524–529. [CrossRef] [PubMed]
Jiang Y. Wagner L. C. (2004). What is learned in spatial contextual cuing—Configuration or individual locations?Perception and Psychophysics,66, 454–463. [CrossRef] [PubMed]
Kunar M. A. Flusberg S. Horowitz T. S. Wolfe J. M. (2007). Does contextual cueing guide the deployment of attention. Journal of Experimental Psychology: Human Perception and Performance,33, 816–828. [CrossRef] [PubMed]
Kunar M. A. Flusberg S. J. Wolfe J. M. (2006). Contextual cueing by global features. Perception & Psychophysics,68, 1204–1216. [CrossRef] [PubMed]
Makovski T. Jiang Y. (2010). Contextual cost: When a visual-search target is not where it should be. [Research Support, N.I.H., Extramural]. Quarterly Journal of Experimental Psychology,63, 216–225. doi:10.1080/17470210903281590. [CrossRef]
Manginelli A. A. Pollmann S. (2009). Misleading contextual cues: How do they affect visual search?Psychological Research,73, 212–221. doi:10.1007/s00426-008-0211-1. [CrossRef] [PubMed]
Olson I. R. Chun M. M. (2002). Perceptual constraints on implicit learning of spatial context. Visual Cognition,9, 273–302. [CrossRef]
Pelli D. G. (1997). The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision,10, 437–442. [CrossRef] [PubMed]
Preston A. R. Gabrieli J. D. E.Dissociation between explicit memory and configural memory in the human medial temporal lobe. Cerebral Cortex,18, 2192–2207. [CrossRef] [PubMed]
Schneider W. Shiffrin R. M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological Review, 84(1), 1–66. [CrossRef]
Shepard R. N. Metzler J. (1971). Mental rotation of three-dimensional objects. Science, New Series,171, 701–703.
Shomstein S. Yantis S. (2004). Configural and contextual prioritization in object-based attention. Psychonomic Bulletin & Review,11, 247–253. [CrossRef] [PubMed]
Song J. H. Jiang Y. (2005). Connecting the past with the present: How do humans match an incoming visual display with visual memory?Journal of Vision, 5(4):4, 322–330, http://www.journalofvision.org/content/5/4/4, doi:10.1167/5.4.4. [PubMed] [Article] [CrossRef]
Westerberg C. E. Miller B. B. Reber P. J. Cohen N. J. Paller K. A. (2011). Neural correlates of contextual cueing are modulated by explicit learning. Neuropsychologia,49, 3439–3447. [CrossRef] [PubMed]
Wolfe J. M. Klempen N. Dahlen K. (2000). Postattentive vision. Journal of Experimental Psychology: Human Perception and Performance,26, 693–716. [CrossRef] [PubMed]
Zellin M. Conci M. von Mühlenen A. Müller H. J. (2011). Two (or three) is one too many: Testing the flexibility of contextual cueing with multiple target locations. [Research Support, Non-U.S. Gov't]. Attention, Perception & Psychophysics,73, 2065–2076. doi:10.3758/s13414-011-0175-x. [CrossRef] [PubMed]
Footnotes
1  All icons were selected from www.softicons.com, available under a Creative Commons Attribution license.
Figure 1
 
Schematic illustration of display layouts and remapping methods. (A) Display layout in landscape mode; each number denotes an individual icon. (B) Portrait display layout obtained by the position-order invariant remapping method; the arrow indicates the icon remapping sequential order from the landscape to portrait mode. (C) Portrait display layout obtained by the global rotation remapping method; the arrow indicates the rotation direction from the landscape to portrait mode. (D) Portrait display layout obtained by the local invariant remapping method; circled regions remain the same between the landscape and portrait mode. (E) Portrait display layout obtained by the central invariant remapping method; circled regions are invariant.
Figure 1
 
Schematic illustration of display layouts and remapping methods. (A) Display layout in landscape mode; each number denotes an individual icon. (B) Portrait display layout obtained by the position-order invariant remapping method; the arrow indicates the icon remapping sequential order from the landscape to portrait mode. (C) Portrait display layout obtained by the global rotation remapping method; the arrow indicates the rotation direction from the landscape to portrait mode. (D) Portrait display layout obtained by the local invariant remapping method; circled regions remain the same between the landscape and portrait mode. (E) Portrait display layout obtained by the central invariant remapping method; circled regions are invariant.
Figure 2
 
Example displays in the experiments. (A) Example of a landscape display. In this example, the “Apple” icon (second row, right-most column) is the search target. (B) Example of a portrait display. In this case, icons are remapped from the landscape mode by keeping the position order constant in the left-to-right and up-to-down manner (Experiment 1). (C) The top overlay for the target icon (a compound letter “T”).
Figure 2
 
Example displays in the experiments. (A) Example of a landscape display. In this example, the “Apple” icon (second row, right-most column) is the search target. (B) Example of a portrait display. In this case, icons are remapped from the landscape mode by keeping the position order constant in the left-to-right and up-to-down manner (Experiment 1). (C) The top overlay for the target icon (a compound letter “T”).
Figure 3
 
Mean correct response times (RTs) as a function of epoch for the learning (Epochs 1–5) and transfer (Epochs 6–9) sessions; for the latter, mean RTs are shown separately for old displays (denoted by diamonds and solid lines) and new displays (denoted by dots and dashed lines). (A) Experiment 1, position-order invariant remapping. (B) Experiment 2, global rotation remapping. (C) Experiment 3, local invariant remapping. (D) Experiment 4, central invariant remapping.
Figure 3
 
Mean correct response times (RTs) as a function of epoch for the learning (Epochs 1–5) and transfer (Epochs 6–9) sessions; for the latter, mean RTs are shown separately for old displays (denoted by diamonds and solid lines) and new displays (denoted by dots and dashed lines). (A) Experiment 1, position-order invariant remapping. (B) Experiment 2, global rotation remapping. (C) Experiment 3, local invariant remapping. (D) Experiment 4, central invariant remapping.
Table 1
 
Mean learning effect in the training sessions and interference by the addition of new displays in the transfer session, for each experiment.
Table 1
 
Mean learning effect in the training sessions and interference by the addition of new displays in the transfer session, for each experiment.
Perceptual learning Interference associated with the presentation of new displays
Experiment Facilitation (ms) ANOVA Cost (ms) ANOVA
1 364 p < 0.01 60 p = 0.08
2 185 p < 0.05 154 p < 0.05
3 385 p < 0.01 36 p = 0.21
4 380 p < 0.01 98 p = 0.11
Table 2
 
Contextual cueing effects in the transfer session. The reported significance values are as follows: *p < 0.05; **p < 0.01; ***p < 0.001.
Table 2
 
Contextual cueing effects in the transfer session. The reported significance values are as follows: *p < 0.05; **p < 0.01; ***p < 0.001.
Experiment Contextual cueing effect (ms) ANOVA test with F value
Average (Epoch 6–9) Epoch 6 Epoch 7 Epoch 8 Epoch 9 Context (old/new) Epoch (6–9) Interaction
1 86 202*** 5 18 120* 9.25* 9.23** 4.49*
2 120 262*** 7 28 197** 12.48** 1.40 8.80***
3 156 223** 50 147* 205** 21.49*** 5.16** 3.07*
4 163 286** 43 142* 190** 22.42*** 5.63** 3.91*
Table 3
 
Results of recognition test. The reported significance values are as follows: *p < 0.05; **p < 0.01.
Table 3
 
Results of recognition test. The reported significance values are as follows: *p < 0.05; **p < 0.01.
Experiment Noticed repetition Explicit learning When (blocks) Hit rates False alarms d′L d′P d′L = d′P
1 90% 60% 6.89 76.3% 38.3% 2.17** 1.17* p = 0.17
2 80% 30% 7.25 72.9% 43.3% 1.33** 1.17* p = 0.68
3 90% 20% 5.85 65.1% 28.3% 1.56 2.02* p = 0.51
4 80% 60% 4.75 80% 28.3% 3.42** 1.52* p < 0.05
×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×