Role of early visual cortex in trans-saccadic memory of object features

Abstract

Early visual cortex (EVC) participates in visual feature memory and the updating of remembered locations across saccades, but its role in the trans-saccadic integration of object features is unknown. We hypothesized that if EVC is involved in updating object features relative to gaze, feature memory should be disrupted when saccades remap an object representation into a simultaneously perturbed EVC site. To test this, we applied transcranial magnetic stimulation (TMS) over functional magnetic resonance imaging–localized EVC clusters corresponding to the bottom left/right visual quadrants (VQs). During experiments, these VQs were probed psychophysically by briefly presenting a central object (Gabor patch) while subjects fixated gaze to the right or left (and above). After a short memory interval, participants were required to detect the relative change in orientation of a re-presented test object at the same spatial location. Participants either sustained fixation during the memory interval (fixation task) or made a horizontal saccade that either maintained or reversed the VQ of the object (saccade task). Three TMS pulses (coinciding with the pre-, peri-, and postsaccade intervals) were applied to the left or right EVC. This had no effect when (a) fixation was maintained, (b) saccades kept the object in the same VQ, or (c) the EVC quadrant corresponding to the first object was stimulated. However, as predicted, TMS reduced performance when saccades (especially larger saccades) crossed the remembered object location and brought it into the VQ corresponding to the TMS site. This suppression effect was statistically significant for leftward saccades and followed a weaker trend for rightward saccades. These causal results are consistent with the idea that EVC is involved in the gaze-centered updating of object features for trans-saccadic memory and perception.

Introduction

Humans typically make three to five saccades per second (Ahissar & Arieli, 2012; Ibbotson & Krekelberg, 2011; Noton & Stark, 1971; Rayner, 1978, 1998; Yarbus, 1967), and visual processing is suppressed during saccades (Matin, 1974; Mitrani, Mateeff, & Yakimoff, 1970), so useful visual information is limited to brief intermittent fixations (Helmholtz, 1963; Prime, Vesia, & Crawford, 2008, 2010, 2011). Yet we are able to perceive the world in a continuous and coherent manner without gaps and delays (Burr & Morrone, 2011; Irwin, 1996; Matin, 1974; Melcher & Colby, 2008; Prime et al., 2011; Tatler & Land, 2011). This requires the active retention of some information in a continuous, updated internal representation of the external world. Psychophysical experiments suggest that humans are able to monitor three to four objects across saccades (Irwin, 1996; Prime et al., 2008; see also Bays & Husain, 2008). This process, known as trans-saccadic perception, consists of three steps: (a) trans-saccadic visual memory, (b) updating this retained information in gaze-centered coordinates, and (c) trans-saccadic integration of the retained and updated information with new visual information (Germeys, De Graef, & Verfaillie, 2002; Irwin, 1996; Melcher & Colby, 2008; Prime et al., 2011). 

Many recent theoretical treatments assume that trans-saccadic perception involves reentrant motor and cognitive signals into extrastriate and early visual cortex (EVC; Cavanagh, Hunt, Afraz, & Rolfs, 2010; Hamker & Zirnsak, 2006; Hamker, Zirnsak, Ziesche, & Lappe, 2011; Melcher & Colby, 2008; Prime, Neimeier, & Crawford, 2006; Prime, Tsotsos, Keith, & Crawford, 2007; Prime et al., 2008, 2011). Consistent with this, functional magnetic resonance imaging (fMRI) studies have implicated EVC in visual memory (Harrison & Tong, 2009), and both fMRI and neurophysiology studies have implicated EVC in the trans-saccadic updating of object locations (Merriam, Genovese, & Colby, 2007; Nakamura & Colby, 2002). Recent transcranial magnetic stimulation (TMS) studies have also investigated the role of EVC in visual memory during gaze fixation (De Weerd et al., 2012; van de Ven, Jacobs, & Sack, 2012; van de Ven & Sack, 2013). One such study implicated EVC in the retention of object shape information, at least with higher memory loads (van de Ven et al., 2012). To our knowledge, however, the role of EVC in the trans-saccadic memory and updating of object features, such as orientation, has not been tested. 

Here, we used fMRI-guided TMS to investigate the functional role of the human EVC in trans-saccadic perception. Subjects compared the orientations of two briefly presented, spatially congruent visual objects, and saccades in the memory interval sometimes caused these object pairs to fall in opposite visual quadrants (VQs). Triple-pulse repetitive TMS (rTMS) was delivered to either side of the EVC in the saccade/memory interval. We expected that this would not be sufficient to influence the retention and comparison of one object pair in the same VQ (van de Ven et al., 2012). However, we hypothesized that TMS might disrupt the gaze-centered updating and integration of visual features across the EVC, particularly when saccades cause the visual coordinates of the remembered object to shift toward the stimulated retinal quadrant. 

Methods and materials

Subjects

Sixteen healthy subjects (seven females; nine males; age range: 19–40; mean age 28.8 years) participated in our preliminary psychophysical experiments (see below) after providing written informed consent. Thirteen of the 16 healthy subjects (four females; nine males; age range: 19–40; mean age 28.8 years) qualified to participate during fMRI scans. Nine of the 13 healthy subjects (two females; seven males; age range: 19–40; mean age 30.7 years) participated in the TMS sessions. One subject's data were excluded from the analysis due to unstable fixation, leaving eight subjects in our final analysis. All subjects had normal or corrected-to-normal vision and no history of neuropsychiatric disorders, according to self-report. All procedures were approved by the York University Human Participants Review Committee. 

Laboratory setup

During psychophysics and TMS sessions, subjects were seated in a dimly lit room with a distance of 51 cm of the eyes from the display screen. A personalized dental impression bar was used to stabilize their head. A customized computer network of three personal computers was used to display the stimulus, record eye-movement data, and record subject response data. Stimuli were presented on a Dell Trinitron P1130 CRT monitor (75 Hz, 1,024 × 768 pixels) with a circular aperture placed over the display area to remove all external orientation cues, such as the ones from the edges of the screen. This circular aperture had a diameter of 32.8° of visual angle. The luminance level for the probes was 30.9 cd/m² (Figure 1). Eye position was monitored during each TMS session using an EyeLink II eye tracker (SR Research Ltd., Ottawa, ON, Canada), which was mounted to the bite bar. 

Figure 1

 

Illustration of the experimental design for the fixation task and saccade task. Subjects were required to make a two-alternative forced-choice response, making a comparison of the orientation of the probe to a previously presented target. Subjects were required to fixate at a fixation point (diameter = 0.16°), presented randomly at 3° or 9° to the right or left of the subjects' head-centered location, which was designated as 0°. This was followed by a target (Gabor patch = 2.9°), presented either in the bottom right or bottom left quadrant of the visual field, relative to fixation. During the fixation task, subjects were required to maintain fixation at the same location such that the following probe appeared in the same retinal location. Subjects were then required to make a response by rotating a mechanical knob to indicate a comparison of the orientation of the probe with the previously presented target. During the saccade task, the second fixation point was presented at a different location. A 10-Hz rTMS train of three pulses was applied 100 ms after the appearance of the second fixation point.

Eye movement recordings and analysis

A nine-point grid of fixation points was used to calibrate the eye tracker before the start of each session for all subjects. Stability of the eye position during fixation and saccade trials was evaluated using a custom program written using Matlab 7.0.0 (The MathWorks Inc., Natick, MA). During the fixation task, eye data were evaluated on the basis of maintaining stable eye position throughout the trial, maximally deviating 1° from the fixation point. During the saccade task, eye data were evaluated based on (a) saccades made within 400 ms of the onset of the second fixation point, and (b) saccades made of appropriate size in comparison to the saccade size required for a given trial (maximally acceptable discrepancy: 1.5°). If these criteria were not met during any given trial, the data from that trial was excluded from the analysis. 

Psychophysical paradigm

Before TMS experiments began, we conducted preliminary psychophysical tests on 16 subjects to determine (a) which subjects could do the task and (b) the best points on their psychophysical curves to test (for maximum statistical power) during the limited amount of time we can apply TMS. Subjects were required to discriminate between pre- and postorientation of an object (a Gabor patch; diameter = 2.9°) during fixation and across saccades (Figure 1). A two-alternative forced-choice procedure was used that required the subject to rotate a mechanical knob in the direction of the perceived orientation change. Each trial began with a circular fixation point (diameter = 0.2°), presented at either 3° or 9° to the right or left from the subjects' head center, which was designated as 0° (see Figure 2 for the detailed combinations). Following a period of 750 ms to allow for fixation, the first oriented Gabor patch was presented for 100 ms either in the bottom right or bottom left VQ, relative to gaze. This was immediately followed by a gray mask for 100 ms in order to reduce external effects of afterimages. 

Figure 2

 

(a) Location of fixation points (−9°, −3°, +3°, +9°) during the fixation task, (b) during the within-hemisphere conditions for the saccade task, and (c) during the across-hemisphere conditions for the saccade task.

During the fixation trials, the second circular fixation point was presented at the same position whereas during the saccade trials the second fixation point was presented at one of the other three possible fixation locations (see Figure 2 for the detailed combinations). As such, during the saccade trials, subjects were required to make a saccade either within the same VQ or from one VQ to another (right to left or left to right VQ). Subjects refixated on the second fixation point for 750 ms, followed by presentation of a probe (Gabor patch; diameter = 2.9°) presented in the same location but in an altered orientation for 100 ms. A second gray mask was presented for 150 ms to reduce afterimages. During a subsequent interstimulus interval of 1000 ms, a white screen was presented. Subjects were required to rotate a mechanical knob with their right hand, in either the clockwise or counterclockwise direction, to indicate the perceived change in orientation of the probe in comparison to the previously presented target. Subjects were asked to make their response as soon as the trial ended and to make their best guess if they were unsure. Accuracy was given a greater emphasis than the speed of response. 

Sixteen subjects were tested on this preliminary no-TMS task. The object rotation was adjusted in 5° steps to obtain a psychometric function (averaged across fixation and saccade trials). During this session, data were collected using a blocked design (eight blocks, four fixation task blocks and four saccade task blocks) with the order counterbalanced. Each block contained 72 trials, yielding a total of 288 fixation trials and 288 saccade trials. Of these subjects, three did not meet our fixation criteria (see above) or perform the orientation discrimination with sufficient proficiency. For the subjects who went on to perform all experiments, we used the individual psychometric functions obtained from this task to set an orientation shift that would obtain a performance level (i.e., mean percentage clockwise responses) between 75% and 85% in the similar TMS experiment. We chose the 75%–85% range to be able to observe both facilitative and suppressive effects of rTMS (i.e., in the two-alternative forced-choice task, 50% clockwise responses represents chance level with 100% representing only clockwise responses). As such, we chose the window of 75%–85% accuracy level. This also served as an exclusion criterion. Subjects who did not reach 75% clockwise responses for the tested range of orientation changes were excluded from the main experiment. 

fMRI localizer task and neuro-navigation

Our main experiment (see next section) required the placement of a TMS coil over the portions of EVC corresponding to the bottom left and bottom right visual hemifields. Therefore, an fMRI localizer task was used to first identify peak EVC activity corresponding to these VQs in the subjects who passed the first stage of exclusion in our psychophysical task. A bifield alternating checkerboard wedge stimulus was used because it is well suited for identification of EVC, particularly V1 (Kraft et al., 2005). Subjects were required to fixate on the center fixation point while checkerboard wedges appeared in the bottom right, bottom left, upper right, and upper left VQs. Subjects were scanned using a 3.0 Tesla Siemens 32-channel head coil whole body scanner at the Neuroimaging Centre at Sherman Health Sciences Research Centre, York University. First, anatomical scans using the MPRAGE sequence, 1 mm³ (isotropic) voxels, were obtained for each subject. Second, four identical scanning runs (256 s) were performed using the EPI sequence (TR = 2 s; TE = 30 s; 29 horizontal or oblique interleaved slices; 3-mm slice thickness with 0.75-mm gap between slices; A → P phase encoding; 64 × 64 in-plane matrix, field of view = 192 mm, 3 × 3 × 3 mm³ voxel size; flip angle of 90°; parallel imaging acceleration factor = 2; bandwidth = 762 Hz/pixel). 

Localizer data were analyzed using BrainVoyager QX 2.10 (Goebel et al., 2006). Motion correction was performed to account for any possible movement of the subject's head in the scanner, thereby preventing misalignment of voxels to the respective brain areas. Functional data (from the four runs) was averaged and overlaid onto the anatomical scan. A stimulation protocol was created for each functional run and was analyzed using the general linear model (single study). The averaged functional data were coregistered with a high-resolution anatomical, a 3-D aligned time course was created for each functional run and analyzed using the general linear model (“multistudy”). The activation threshold was increased to determine the voxel with peak activation within the retinal quadrant (Figure 3b; Table 1). This information was later used to navigate the TMS coil using the BrainSight 2.0 neuro-navigation system (Rogue Research Inc., Montreal, QC, Canada) during the TMS sessions. A calibration was performed with each subject in order to establish the spatial relationship between the anatomical image data and markers attached to the subject's head using safety goggles during the experiment. Registration was conducted by selecting common points on the image data and the subject (nasion, tip of nose, right ear, and left ear [upper part of the tragus]). These homologous point pairs are used by the BrainSight 2.0 neuro-navigation system to calculate the spatial transformation from the subject's head to his or her image data (Rogue Research). 

Figure 3

 

(a) Localizer stimulus presented to subjects during the fMRI scans. Alternating checkerboard wedges were presented as eight trials of 32 s for each run. A total of four runs (256 s each) were conducted for each subject, and the data were averaged for analysis purposes. (b) Functional localizer analysis for a representative subject, conducted using BrainVoyager QX 2.10, illustrating clusters of activation corresponding to the various quadrants of the visual field in the sagittal and transverse planes.

Table 1

 

Talairach coordinates of the peak activation voxel and volume of activation clusters in the early visual cortex of the right and left hemispheres, determined via functional localizers and analysis using BrainVoyager. Notes: Measurements of the target depths, measured from the outer skull surface to the point of stimulation (in millimeters), using BrainSight 2.0.

 

rTMS experiment

Nine subjects passed the psychophysical assessment stage, had EVC retinal quadrants identified using fMRI, and were willing to continue with the TMS experiment. Subjects were again screened for health and safety at the start and end of each TMS session. This experiment utilized the same psychological paradigm as that shown in Figure 1 with the additional application of triple-pulse rTMS. The timing of the TMS pulses, starting at 100 ms from the onset of the second fixation point (Figure 4) with a 100-ms interval in between, was selected to ensure that we could capture the pre-, peri-, and postsaccade timing with each one of the pulses (Figure 4). We used three pulses (10 Hz triple-pulse train) as opposed to single pulses used in our previous studies of the frontal and parietal eye fields (Prime et al., 2008, 2010) because previous experiments suggested that the EVC is generally more resistant to TMS effects (van de Ven et al., 2012) and because we could not predict which portion of the perisaccadic interval might be most sensitive to TMS in this sensory area. Importantly, our train of three pulses was likely followed by inhibition lasting 200–300 ms (Moliadze et al., 2003; Nakamura et al., 2014), which meant that the effect of our stimulation could be classified as perisaccadic. We applied rTMS unilaterally with the aim of stimulating the EVC to modify transmission of visual input in a field-specific fashion. The TMS coil was positioned based on the retinotopically defined EVC quadrants (corresponding to the bottom right and bottom left VQs) defined above. We did not use a separate TMS site as control, as we have done in other studies, because this study was designed to compare the effects of TMS over left versus right EVC across specific symmetric behavioral conditions (see next section), and these two sites would clearly provide better controls for each other than another arbitrarily chosen site. The coil was held with the handle facing down, rotated 45° away from the midline (clockwise for left EVC and counterclockwise for right EVC); the coil surface was held perpendicular to the scalp as guided by the BrainSight system. We aimed to alter cortical excitability of EVC by using TMS. The timing of the TMS pulses (triple pulse), starting at 100 ms from the onset of the second fixation point (Figure 1) with a 100-ms interval in between was selected to ensure that we could capture the interval from before saccade onset until after saccade completion timings with each one of the pulses (Figure 4). A comparison of the frequency distribution of saccade onset and saccade endpoint with TMS pulse timings illustrated that a majority of saccades were found to be initiated between the 201- and 220-ms and completed between the 221- and 260-ms time windows. 

Figure 4

 

Comparison of the frequency distribution of saccade onset (dark gray) and saccade endpoint (light gray) with TMS pulse timings. Number of trials for each discrete range of 20 ms, starting at 100 ms to 400 ms are illustrated. Majority of saccades were found to be initiated between the 201- and 220-ms and completed between the 221- and 260-ms time windows.

During the TMS sessions, rTMS was applied to retinotopically defined regions of the right and left EVC. The no TMS condition was used as a baseline control to determine how accurate subjects were in detecting orientation change in the absence of interfering effects of TMS. TMS (right and left EVC) and no-TMS baseline control trials for the fixation and saccade tasks were presented in a block design in a R-L-N-N-L-R, R-N-L-L-N-R, L-R-N-N-R-L, L-N-R-R-N-L, N-R-L-L-R-N, or N-L-R-R-L-N design for the left EVC (L), right EVC (R), and no-TMS (N) conditions. Furthermore, for each TMS site, an FS-FS-FS-SF-SF-SF or SF-SF-SF-FS-FS-FS design was used for the fixation (F) and saccade (S) conditions. These orders were counterbalanced across subjects. Each block consisted of 80 trials, yielding a total of 960 trials (480 fixation trials and 480 saccade trials) for the entire experiment for each subject. The TMS and no-TMS trials were presented in separate blocks to prevent any anticipatory effects. 

All subjects received stimulation of 60% stimulator output intensity. Expressed as a percentage of resting motor threshold (RMT), this was equal to a mean of 98.7% across subjects (range 85.7%–111.1% with no clear outliers; Table 2). RMT was recorded for all subjects on the basis of a visual finger and/or hand movement. RMT was defined as the lowest single-pulse TMS intensity over the hand area of the primary motor cortex that resulted in a visible finger movement in the right hand in three of the six cases tested (Rossini et al., 1994). We initially positioned the coil over the anatomical “hand knob area” (Yousry et al., 1997) of the left hemisphere and repositioned it as well as altered stimulator intensity to find an optimal “hot spot.” At the end of each TMS experiment session, subjects were asked to report if they saw phosphenes during stimulation; none of the subjects reported seeing phosphenes. 

Table 2

 

Resting motor threshold values recorded for all subjects. Notes: All subjects received TMS at 60% stimulator output intensity. The values in column 3 express this in terms of the resting motor threshold recorded for these subjects from left M1 stimulation.

 

Specific hypotheses and analysis

Figure 2 shows the different spatial combinations of trials in our study and how we grouped our data for analysis in terms of fixation trials (Figure 2a), trials in which saccades stayed on the same side of midline (Figure 2b), and trials in which saccades crossed midline (Figure 2c). According to previous fMRI and neurophysiological studies, the trials in Figure 2b should be associated with “remapping” retained information within one hemisphere whereas the trials in Figure 2c should involve remapping across hemispheres (Berman, Heiser, Dunn, Saunders, & Colby, 2007; Colby et al., 2005; Heiser & Colby, 2005; Medendorp et al., 2005; Merriam & Colby, 2005; Merriam, Genovese, & Colby, 2003; Merriam et al., 2007). Further, a previous study found that TMS over the EVC does not influence memory of a single visual object during fixation (van de Ven et al., 2012), but our previous results (with other brain areas) suggest that TMS can have stronger suppressive effects on visual memory during saccade than fixation, presumably because saccade-dependent remapping is a more labile internally driven process than stimulus-driven perception (Prime et al., 2011), and neurophysiological remapping signals are much less robust compared to visual responses (Duhamel, Colby, & Goldberg, 1992; Subramanian & Colby, 2014). Likewise, we expected that TMS would induce larger effects for larger saccades because they last longer and are associated with more internal “noise” (Abrams, Meyer, & Kornblum, 1989; Byrne & Crawford, 2010; Prime et al., 2007). 

Based on these assumptions, we developed the following hypotheses: (a) as observed previously for a single remembered visual stimulus (van de Ven et al., 2012), TMS would have a minimal effect in the fixation task whether TMS is applied to the retinal quadrant of perception or the opposite quadrant (Figure 2c, i.e., whenever EVC activity is only triggered by an external object); (b) TMS would maximally affect performance in situations in which visual information is expected to be remapped across hemispheres/visual fields into the stimulated EVC quadrant (i.e., TMS corresponding to the VQ contralateral to the one in which the object was viewed; Figure 2c). We expected this because (a) this condition would be associated with the largest saccades and (b) in this situation, TMS would specifically isolate an internally driven signal, independent of the original sensory input in the opposite hemisphere (see Figure 8a for a graphic illustration). Our experiment and analysis was primarily designed to test these two hypotheses, using topographically identified regions of left EVC and right EVC as controls for each other. 

Based on the assumptions listed above, we were less certain about what to predict during situations that are intermediate between those two extremes, i.e., (a) when TMS is applied to an EVC quadrant during saccades that should produce remapping of features within this visual quadrant (Figure 2b) or (b) when TMS is applied to the “perceiving” EVC quadrant during saccades that should remap this information toward the opposite EVC hemisphere. In the first situation, saccades are smaller, and even if some remapping errors were produced, it might be sufficient for subjects to solve the task based on the initially perceived position of the stimulus in that quadrant. In both cases, one would presumably be targeting some mixture of remapping signals (that might be affected by EVC TMS) and ordinary visual memory signals (which are not affected by EVC TMS), so it is difficult to make a clear prediction. 

Offline analysis showed that one of our nine subjects did not meet our fixation criteria and was excluded from further analysis. A paired samples t test was conducted to compare baseline performance during the no-TMS control trials for the fixation task versus the saccade task. For the fixation task (Figure 2a), trials were grouped based on the object maintained within the right VQ or within the left VQ, and performance was assessed based on the stimulation conditions (no TMS, left EVC, and right EVC). Based on previous studies, we expected to observe little TMS effects in the fixation task. To explore the general effects of stimulation conditions on performance in the fixation task, we thus conducted a TMS condition (left EVC, right EVC, no TMS) × target VQ (left and right) repeated-measures ANOVA. A similar analysis was used for the saccade task when saccades maintained the object within the same visual hemifield (Figure 2b). For this condition, the literature did not allow us to make strong predictions with respect to the TMS effects, and we thus opted to conduct an exploratory ANOVA. 

We predicted a hemisphere dependence of TMS effects for saccades that crossed the midline (Figure 2c), which caused a change in the location of the object (from left VQ to right VQ and from right VQ to left VQ), resulting in remapping in the opposite hemisphere. This prediction is most adequately and directly tested using a paired samples t test. For these types of trials, we further explored the additional effects of saccade size using a stimulation condition (left EVC, right EVC, no TMS) × saccade size (small, medium, large) repeated-measures ANOVA (with pairwise comparisons using Bonferroni corrections). Finally, to directly assess any possible confounding effect of saccade end points (near vs. far from presented object), we employed paired samples t tests. Results were also summarized based on relative performance (postsaccadic VQ to presaccadic VQ) for saccade direction and saccade size to visualize the effects of stimulation over the perceived VQ versus the opposite (remapped) VQ. 

Finally, regarding statistics, it has been reported that five participants may suffice when using an fMRI localizer (Sack et al., 2009) whereas we analyzed data from eight. However, we acknowledge the possibility of false negatives in such studies so will mainly focus on our significant, hypothesis-driven results without either ignoring or overstating the negative results. 

Results

fMRI localizers

Example localizer data are illustrated in Figure 3b. These typically revealed several areas of activation, including two major clusters (average left EVC: x = −13.6 ± 4.7, y = −89.3 ± 4.9, z = −2.1 ± 9.4, average depth under the skull = 17.8 mm; average right EVC: x = 7.4 ± 2.8; y = −90.3 ± 6.1; z = −2.5 ± 9.7, average depth under the skull = 17.4 mm), corresponding to the posterior tip of the occipital pole. Talairach coordinates of the peak activation voxels corresponding to the bottom right and bottom left VQs are shown in Table 1. We used the BrainSight 2.0 neuro-navigation system to map the fMRI coordinates to TMS coordinates on an individual basis in their native space. These coordinates were used to navigate the TMS coil using the BrainSight 2.0 neuro-navigation system (Rogue Research) during the TMS sessions. A comparison of the targeted coordinates in Talairach space (see Table 1) with literature values (average left EVC: x = −13, y = −63, z = 3; average right EVC: x = 9; y = −67; z = 5; Dougherty et al., 2003), and the volume of our clusters was (see Table 1) confirmed that ipsilateral V1 was the main target of the TMS effect although the spread of TMS likely influenced the same VQ in area V2 (Dougherty et al., 2003) and may have influenced the corresponding structures in the opposite EVC to a lesser degree. Based on these data and the well-known topography of this area (Dougherty et al., 2003; Merriam & Colby, 2005; Merriam et al., 2003, 2007), we assumed that TMS over these areas would primarily influence vision in the opposite hemifield, and thus, the left EVC and right EVC would serve as control sites for each other in the following TMS experiment. We measured the target depths relative to the outer skull surface (the point of stimulation; in millimeters), using BrainSight 2.0 (see Table 1). A paired samples t test revealed no significant difference in the target site depths between the right and left hemispheres, t(7) = −0.63, p = 0.55, thus ensuring that any hemisphere dependence of the TMS effect is not due to an asymmetry in the target depths. 

Baseline (no TMS) psychophysical performance

Performance in the no TMS trials (obtained during the TMS experiment) across all subjects for the fixation task was 84.7 ± 3.4% and for the saccade task was 73.9 ± 3.8%. Subject's performance in the two task types was significantly different, t(7) = 3.59; p = 0.01. However, there was no difference in performance between the object presented in the left or right VQ in the fixation task, t(7) = 1.35; p = 0.22 (Figure 5a, blue bars), when saccades kept the object in the same VQ during the saccade task, t(7) = −0.09; p = 0.93 (Figure 5b, blue bars) or when saccades crossed midline and thus changed the VQ in which the object was presented, t(7) = −0.80, p = 0.45 (Figure 6, blue bars). Based on these control findings, we did not make an attempt to conduct comparisons between tasks in our TMS data but instead focused on the within-task comparisons required to test our hypotheses (see Methods and materials), in particular, comparisons between left EVC and right EVC TMS effects (see below). 

Figure 5

 

Results of the no TMS baseline control (blue), left EVC TMS (green), and right EVC TMS (yellow) stimulation conditions (a) during the fixation task and (b) during the saccade task when the saccades did not cross midline. In these trials, the location of the object is maintained within the right or left VQ.

Figure 6

 

Results of the no TMS baseline control (blue), left EVC TMS (green), and right EVC TMS (yellow) stimulation conditions during the saccade task in which the saccades crossed midline and the object was remapped across hemispheres. Eye movements resulted in the object reversing the hemifield (bottom right to bottom left or vice versa).

TMS during fixation task

Figure 5a compares the performance for the no TMS baseline control (blue bars), left EVC (green bars), and right EVC TMS (yellow bars) conditions across all subjects for the fixation task with data sorted according to the hemifield of object appearance (right VQ vs. left VQ; Figure 2a). In this task (where, of course, object location was maintained within the same visual hemifield), there was no significant effect of stimulation condition, F(2, 14) = 0.15; p = 0.86; Display Formula = 0.02. TMS effects also did not depend on the visual quadrant, F(1, 7) = 0.35; p = 0.57; Display Formula = 0.05), nor was there any differential effect of stimulation condition for the different quadrants: stimulation condition × visual quadrant interaction, F(2, 14) = 1.40, p = 0.28; Display Formula = 0.17. In summary, TMS over EVC had no effect in our fixation task in agreement with previous findings that combined TMS over EVC and memory of a single visual object (van de Ven et al., 2012).  

TMS during saccades that did not cross midline

Saccade onset was found to be between 201 and 220 ms, and saccade offset was determined to be between 221 and 260 ms in a majority of trials (Figure 4). Figure 5b compares the performance for the no TMS baseline control, left EVC, and right EVC TMS conditions across all subjects for the saccade task in which eye position did not cross midline (Figure 2b). Just as for the fixation task, for this task there were no significant effects of simulation condition, F(2, 14) = 0.30, p = 0.74; Display Formula = 0.04, and VQ, F(1, 7) = 0.15, p = 0.91; Display Formula = 0.002, and the interaction of these effects also was not significant, F(2, 14) = 0.91, p = 0.43; Display Formula = 0.12. In summary, even in the saccade task, TMS had no effect when the object was both perceived and retained in the same VQ.  

TMS during saccades that crossed midline

We predicted TMS effects to be most distinct and hemisphere-dependent for saccades that crossed midline, causing the location of the object to be remapped in the opposite hemisphere (Figure 2c). We indeed observed a suppression of performance during TMS over the EVC quadrant corresponding to final (remembered and remapped) visual object, compared to TMS over the opposite (perceiving) hemisphere (Figure 6). A significant suppressive effect of TMS was found for the left EVC (in comparison to the contralateral right EVC TMS) when saccades caused the object to shift from the left VQ to the right VQ, t(7) = −2.46; p = 0.04 (Figure 6). Right EVC TMS (compared to left EVC TMS) when saccades were made from the right VQ to the left VQ also showed a decrease in performance, but this was not significant, t(7) = 0.70; p = 0.51 (Figure 6). All of the saccades included within this analysis occurred within the estimated period of the TMS effect (Moliadze et al., 2003, Nakamura et al., 2014), but some (12.9%) started just after the last TMS pulse. To be sure that this did not influence our results, we recalculated the data after removing these trials and obtained similar results for both leftward saccades, t(7) = −2.463, p = 0.043, and rightward saccades, t(7) = 0.616, p = 0.558. Thus, TMS suppressed performance when applied to the side of EVC that did not “perceive” the visual object but would be expected to be activated by an internal remapping process. 

TMS during saccades of different sizes that crossed midline

A previous study found that performance in a somewhat similar task decreased as a function of saccade size (Prime et al., 2007), so we checked to see if the same occurred in our no TMS and/or TMS data. Saccades that crossed midline were grouped into small (6°), medium (12°), and large (18°) for no TMS, left EVC TMS, and right EVC TMS data (Figure 7). This revealed no progressive drop in performance for no TMS, but a clear drop in performance as a function of saccade size for the TMS conditions (especially over the left EVC). 

Figure 7

 

Results of the no TMS baseline control (blue), left EVC TMS (green), and right EVC TMS (yellow) stimulation conditions for the saccade task during which the saccades crossed midline, analyzed on the basis of saccade size (small = 6°, medium = 12°, large = 18°).

Figure 8

 

Hypothesis and summary of main results for the key EVC TMS/saccade across midline tasks. Top row: hypothesis: TMS over EVC interrupts remapping of signals into the corresponding visual field (VF). (a) Leftward saccade reverses internal representation of object from left to right visual field, so left EVC TMS should interfere with remapped memory. Lower row: Confirmation of hypothesis and summary of results using plots of performance (relative percentage correct calculated as postsaccadic VF − presaccadic VF) during EVC TMS corresponding to the “postsaccadic” visual field, minus the control site (TMS over opposite EVC, i.e., the “outgoing” visual field), done separately for each saccade amplitude. The data correspond to the situations shown in the upper row, so in (b) the “postsaccadic” site is the left EVC, and in (c) the “postsaccadic” site is the right EVC. In each case, as predicted by the hypothesis, there is a reduction in performance (negative-going bars) relative to the opposite control site, and also this effect increases with saccade size. (See Results for statistical analysis of the results summarized here).

A 3 × 3 repeated-measures ANOVA with stimulation conditions (no TMS, left EVC, and right EVC) and saccade sizes (small, medium, and large) showed a significant main effect of saccade size, F(2, 14) = 4.70; p = 0.03; Display Formula = 0.40, and a significant interaction between saccade size and stimulation site, F(4, 28) = 3.41; p = 0.02; Display Formula = 0.33. Post hoc comparisons of the interaction effects revealed significant differences between small versus medium (p = 0.01) and medium versus large saccades (p = 0.03) for the left EVC TMS and between right and left EVC TMS (p = 0.02) for the large saccade sizes.  

Because these different saccade sizes tended to correlate with different saccade end points, a further analysis was conducted to test the effects of saccade end point (near or far in relation to the object). There was no significant difference for the medium saccade sizes, resulting in the final eye position near versus far away from the object, t(7) = 1.00, p = 0.35; between trials with medium and small saccades, resulting in the final eye position near the object, t(7) = −1.74, p = 0.13; and no significant difference between trials with medium saccades resulting in the final eye position far away from the object, t(7) = 1.61, p = 0.15. Thus, there was no significant influence of eye position on the TMS effect although one cannot exclude the possibility that some of these modulations would become significant with a larger data set. To summarize this section, performance was significantly reduced as a function of saccade size for saccades that crossed midline during TMS trials (especially for the left EVC TMS condition). 

Discussion

Our findings are the first to implicate human EVC in the trans-saccadic perception of visual feature information (specifically object orientation) across saccades. It has long been known that the EVC is rich in object orientation information and represents contralateral space in a retinotopic fashion (Hubel & Wiesel, 1968; Spear & Baumann, 1979). More recent studies have suggested that the striate cortex has access to representations in visual short-term memory storage (Harrison & Tong, 2009; van de Ven et al., 2012) and a role in remapping of visual targets during saccades (Merriam et al., 2007; Nakamura & Colby, 2002). However, little research has been conducted to examine the functional role of the EVC in the integration of visual feature information across saccades. Our results (summarized graphically in Figure 8b, c) suggest that memory of object orientation was inhibited when saccades across midline brought the gaze-centered location of the remembered object in line with the stimulated retinal quadrant and that this effect increases with saccade size. This inhibition was only statistically significant for leftward saccades but followed a similar trend in rightward saccades (Figure 8). Importantly, it is possible that the two hemispheres (i.e., the EVCs) might have similar functions but different TMS thresholds for some unknown reason, which could contribute to this asymmetry. Nonetheless, our results implicate EVC in remapping the object attributes and suggest that the neural representations for trans-saccadic memory are (at least at the early stages) retinotopically defined. 

Mechanism and interpretation of TMS action

Consistent with the results of a previous study (van de Ven et al., 2012), we did not find any effect of TMS on short-term memory of a single object. However, we did find an effect in situations in which one would expect the remembered object to be remapped contralaterally into the magnetically stimulated EVC/quadrant (Figure 8a). Again, this effect was only statistically significant for leftward saccades, and was much weaker overall for rightward saccades (Figure 6). The mathematical reason behind this can be appreciated by the summary in Figure 8, which shows normalized TMS performance (“remapping” hemisphere to “perceiving” hemisphere), sorted first by direction (Figure 8b, c) and then also by saccade size. Here, very similar saccade amplitude–dependent trends are visible for both saccade directions, but performance relative to the control site is shifted upward (more positive) for the right EVC/rightward saccades (Figure 8c). It is thus possible that the statistics for these rightward saccades might be a “false negative,” but given that the p value for this result was more than an order of magnitude greater than the standard 0.05, we cannot make such a claim based on our current data set. Previously, we also observed hemisphere-specific effects during TMS over the parietal cortex in a similar task with multiple stimuli (Prime et al., 2008). Not enough is known at this time about the mechanisms of action of TMS to know for certain if these asymmetries are related to these mechanisms or intrinsic brain functions. It may be that TMS suppressed feature remapping here by decreasing signal strength, by overwriting the neural representation of memory trace in the visual cortex, or (as we believe) by injecting “noise” into local cortical signals (Harris, Clifford, & Miniussi, 2008; van de Ven et al., 2012; Vesia, Prime, Yan, Sergio, & Crawford, 2010). In any case, our finding is consistent with both the general notion that trans-saccadic memory, updating, and integration involve a complex interplay between sensory, cognitive, and motor signals and the specific notion that this involves recurrent feedback to the EVC (Cavanagh et al., 2010; Hamker & Zirnsak, 2006; Hamker et al., 2011; McMains & Kastner, 2011; Melcher & Colby, 2008; Prime et al., 2006; Prime et al., 2007; Prime et al., 2008, 2011). 

We did not find any significant effects of TMS in situations that would involve intrahemispheric (i.e., within VQ) remapping, perhaps because the neural mechanisms for within-hemisphere remapping are fundamentally different and more robust than those for across-hemiphere remapping (Berman et al., 2007). However, because these trials involved smaller shifts in retinal stimulus position, it may be that subjects were able to rely more on their original perception or tolerate small remapping errors and still solve the task. Finally, because these trials were generally associated with smaller saccades, and it was the largest saccades that mostly drove the effect for our cross-field results, this might in part be explained by finding that suppression increased with saccade size. Larger saccades take longer to produce (Abrams et al., 1989), so this would allow more time for TMS to influence the signals (saccade efference copies and other computations) associated with remapping (Keith, Blohm & Crawford, 2010). Larger saccades have also been shown to reduce performance during egocentric updating and trans-saccadic integration tasks in the absence of TMS, presumably due to noisy internal signals (Byrne & Crawford, 2010; Prime et al., 2007). This was not observed in our control data here, but it is possible that such internal noise interacted with TMS-induced noise to produce our amplitude-dependent effects. The combination of these various effects leads to the overall pattern summarized in Figure 8a in which the effect of TMS on the postsaccadic EVC increased with saccade size. Taking all these factors into account, it is possible that TMS had some effect on intrahemispheric remapping that was not measureable above background noise, but again, these results did not approach significance, so we cannot make such a claim based on our data set. 

EVC and the “trans-saccadic integration network”

Given that the neural noise induced by TMS might influence both EVC and the structures that it connects to, we cannot be certain that our effects were caused primarily at the site of stimulation. Areas V2 and V3 (Nakamura & Colby, 2002), V4 (Merriam et al., 2007), LIP (Duhamel et al., 1992; Goldberg & Bruce, 1990; Gottlieb, Kusunoki, & Goldberg, 1998; Heiser, Berman, Saunders, & Colby, 2005; Kusunoki, Gottlieb, & Goldberg, 2000; Li & Andersen, 2001), frontal eye fields (FEF; Kastner et al., 2007), and superior colliculus (SC; Nakamura & Colby, 2002; Umeno & Goldberg, 1997; Walker, Fitzgibbon, & Goldberg, 1995) have been observed to be involved in spatial remapping. However, when brain function is considered from a network dynamics perspective, this is not a clear distinction. For example, this would still indicate that proper EVC function is necessary for optimal function of this entire network and that any pathology leading to physiological noise within EVC would be expected to have similar effects. 

It is also likely that different components of this network fulfill different functions. For example, Prime et al. (2008, 2010) investigated the role of the parietal eye fields (PEF) and FEF in trans-saccadic integration using a similar task except that those experiments utilized multiple saccade directions and a range of different object set sizes. As in the current study, these studies showed greater effects during saccades compared to fixation, and as in van de Ven et al. (2012) these effects increased with set size. But unlike the current study, the TMS effect was independent of saccade direction and visual object direction. Thus, Prime et al. (2006, 2007) interpreted their results as a disruption of the saccade efference copy used to drive the remapping of the object features. 

In contrast, the effects observed in the current study were highly dependent on both saccade direction and visual target direction, consistent with storage of information in retinotopically lateralized coordinates. This is consistent with models that give EVC a role in the early aspects of the visual memory storage mechanism and make it the target of recurrent feedback from the saccade system (Bullier, 2001; de Graaf, Goebel, & Sack, 2012; Hochstein & Ahissar, 2002; Lamme & Roelfsema, 2000; Prime et al., 2008, 2010, 2011; Ro, Breitmeyer, Burton, Singhal, & Lane, 2003). However, this does contradict a role for higher-level visual cortex in trans-saccadic memory and perception. High-level visual cortex is also dominantly retinotopic (Golomb & Kanwisher, 2012) but under some circumstances might show spatiotopic organization and/or gaze-centered remapping (Burr & Morrone, 2011; Crespi et al., 2011). For example, a recent neurophysiological study in the monkey showed that a modest amount of object information is retained and remapped across saccades in lateral intraparietal cortex (Subramanian & Colby, 2014). This might explain why TMS over human PEF influences the trans-saccadic memory of a single object (Prime et al., 2008) whereas TMS over FEF only has an influence when multiple target locations must be remembered (Prime et al., 2010)—perhaps a more pure “motor” effect. 

Alternative theories and explanations

An alternate theoretical explanation for our result is that TMS only interrupted the updating of object location, and this is somehow linked indirectly to object orientation (Cavanagh et al., 2010; Crawford, 1997). Such mechanisms might play a role in higher-level aspects of trans-saccadic integration but are unlikely to explain the current “low-level” result. First, as noted above, this would likely produce more general spatial effects as observed for FEF and PEF (Prime et al., 2011). Second, in principle, our task (making comparisons between a single object pairing) can be solved without spatial information, so it is unlikely that a purely spatial mechanism would explain the current result. Finally, there is already precedent for feature remapping in other brain areas (Greenlee, Magnussen, & Reinvang, 2000; Subramanian & Colby, 2014). 

Another possibility is that our TMS effects were actually caused by interfering with perception of the test object rather than memory of the initial object. This is also unlikely because TMS only had an effect when the remembered object appeared on the VQ opposite to the TMS site, and the effect was otherwise unrelated to the side of the test object during either fixation or saccades (i.e., TMS had no effect when the perceived and remembered saccades were both on the contralateral side or both on the ipsilateral side to TMS). Further, it is likely that TMS has a more disruptive effect on memory (perhaps because it must be maintained by internal activity) than it does on perception, with which extrinsic information likely overrides the effects of TMS (Lee & van Donkelaar, 2002; Melcher, 2009; Melcher & Colby, 2008). Indeed, in our hands, TMS has been most effective in tasks that involved short-term memory and/or planning of a future movement in the absence of sensory feedback (Prime et al., 2011; Vesia et al., 2010; but see Dessing, Vesia, & Crawford, 2013). 

Conclusions

Our study is consistent with the idea that a significant component of trans-saccadic memory and integration occurs (at least at early levels of the visual system) in gaze-centered coordinates, and involves the remapping of signals within these coordinates during saccades (Figure 8). Second, it supports the notion that trans-saccadic memory involves additional computations to visual working memory, at the least the saccade-dependent signals required for remapping (Merriam & Colby, 2005; Merriam et al., 2003, 2007). Finally, it is consistent with the notion that EVC (or closely associated occipital structures) are involved in spatial remapping across saccades, presumably with the aid of recurrent connections related to attention and saccades (Merriam et al., 2007). However, based on our data, we cannot say whether the TMS effects reported here reflect local EVC processes versus the transmission of disrupted EVC signals onto other brain areas. It is likely that the EVC is only one part of the visual memory storage system during saccades. Other structures, including parietal, temporal, and frontal cortex (as well as subcortical structures) are also likely to be involved, perhaps depending on the task (de Graaf et al., 2012; Prime et al., 2007; Prime et al., 2010, 2011; Thielscher, Reichenbach, Ugurbil, & Uludag, 2010). 

Acknowledgments

This project was supported by a grant from the National Science and Engineering Research Council of Canada. J. D. Crawford is supported by a Canada Research Chair Award. The authors thank S. Sun for computer programming support. Conflict of interest: The authors declare that there are no conflicts of interest. 

Commercial relationships: none. 

Corresponding author: Douglas Crawford. 

Email: [e-mail]. 

Address: Centre for Vision Research, Toronto, ON, Canada.