Characterizing eye movement behaviors and kinematics of non-human primates during virtual navigation tasks

Abstract

Virtual environments (VE) allow testing complex behaviors in naturalistic settings by combining highly controlled visual stimuli with spatial navigation and other cognitive tasks. They also allow for the recording of eye movements using high-precision eye tracking techniques, which is important in electrophysiological studies examining the response properties of neurons in visual areas of nonhuman primates. However, during virtual navigation, the pattern of retinal stimulation can be highly dynamic which may influence eye movements. Here we examine whether and how eye movement patterns change as a function of dynamic visual stimulation during virtual navigation tasks, relative to standard oculomotor tasks. We trained two rhesus macaques to use a joystick to navigate in a VE to complete two tasks. To contrast VE behavior with classic measurements, the monkeys also performed a simple Cued Saccade task. We used a robust algorithm for rapid classification of saccades, fixations, and smooth pursuits. We then analyzed the kinematics of saccades during all tasks, and specifically during different phases of the VE tasks. We found that fixation to smooth pursuit ratios were smaller in VE tasks (4:5) compared to the Cued Saccade task (7:1), reflecting a more intensive use of smooth pursuit to foveate targets in VE than in a standard visually guided saccade task or during spontaneous fixations. Saccades made to rewarded targets (exploitation) tended to have increased peak velocities compared to saccades made to unrewarded objects (exploration). VE exploitation saccades were 6% slower than saccades to discrete targets in the Cued Saccade task. Virtual environments represent a technological advance in experimental design for nonhuman primates. Here we provide a framework to study the ways that eye movements change between and within static and dynamic displays.

Introduction

Virtual environments (VE) are increasingly relied upon for immersive yet highly controlled experiments across many research domains. This is because they allow for experimenter control of stimuli and their timing while giving subjects a sense of agency and exploration. They also allow recording of behavioral outcomes precisely synchronized with events in the VE, which is necessary for computing measurements of performance (Bohil, Alicea, & Biocca, 2011; Washburn & Astur, 2003). Both humans and nonhuman primate eye movements are a critical element of environmental exploration to ensure optimal survival. They have been extensively studied in their involvement in vision (Haarmeier & Thier, 1999; Tatler, Hayhoe, Land, & Ballard, 2010), cognition (Di Stasi et al., 2010) and motor control (Watanabe & Munoz, 2011). However, most of these studies have used two-dimensional displays in which most objects and the background remain stationary, requiring subjects to respond with a single type of eye movement behavior (e.g., saccades, smooth pursuits, or fixations). During virtual navigation, objects and environmental features that are common targets of eye movements become dynamic as one moves about them. This creates a nontrivial challenge of determining when subjects are foveating an object, for how long, and how they respond to the dynamics of a scene. Whereas studies in VE have led to insights into spatial working memory (De Lillo & James, 2012) and scene memory (Kit et al., 2014), as well as paradigms to interrogate hippocampal activity in humans (Miller et al., 2013) and nonhuman primates (Hori et al., 2005; Wirth, Baraduc, Planté, Pinède, & Duhamel, 2017), there has not yet been a thorough investigation of the eye movement behaviors in VE. It is also unclear how such behavior compares to that of classical tasks used in vision labs. 

Eye movement behaviors can be separated into two broad categories: foveations and saccades ( for review, see Kowler, 2011; Westheimer, 1954). Foveations occur when the fovea, the high acuity part of the retina, is focused on an object in space. If the head and the object are static, this is called a fixation; if the object is moving, then for the eye to track it, a smooth pursuit is executed. If the head is moving, the vestibulo-ocular reflex (VOR) allows the eye to foveate a target by compensating for the movement. These are information gathering behaviors, and their durations are influenced by the demand for information at the target of foveation (Galley, Betz, & Biniossek, 2015). A study has found that there are differences in the time needed to gather information during a fixation or pursuit (Schütz, Braun, & Gegenfurtner, 2009); however this study used a letter discrimination task, which is likely more difficult than the current task. This finding could mean that smooth pursuits might not be as efficient at gathering information as fixations, as another study found that human foveations are shorter on static stimuli than on movies (Dorr, Martinetz, Gegenfurtner, & Barth, 2010). If this relationship holds true in virtual environments, separation of foveation type will be critical when interpreting foveation duration within the virtual reality tasks. 

Saccades make up the second broad category of eye movements. Saccades are ballistic movements that change the foveation target. The relationship between the size (amplitude) of the saccade and the peak velocity of the eye rotation towards the target is called the saccadic main sequence (Bahill, Clark, & Stark, 1975). This relationship is very well characterized; the larger the saccade, the higher the peak velocity (until a velocity ceiling) and the longer the duration (Fuchs, 1967). This relationship can be affected by physical factors, such as the position of the eye and the direction of the saccade (Becker, 1989), by the presence and size of a target (Edelman & Goldberg, 2003), and its value (Bendiksby & Platt, 2006). In humans, cognitive factors such as alertness of the subject have also been shown to modify the main sequence (Di Stasi, Catena, Cañas, Macknik, & Martinez-Conde, 2013; Galley, 1989). VE navigation could modulate engagement or alertness, or show an interaction with task demands as well. 

With the effects of task engagement due to navigation and the dynamic nature of the visual stimuli in VEs, it is not known how eye movement type, proportion, duration, or kinematics could be modulated. Furthermore, the effect of specific cognitive demands within VEs on these parameters is also unknown. To analyze the effect of VE navigation tasks on eye movements, we trained two monkeys on two tasks embedded into a common virtual environment: a Learning task and a Foraging task. To determine whether eye movements in these VE tasks differed from classical visuomotor tasks, we also trained these subjects to complete a Cued Saccade task. We recorded eye movements in each of these tasks and categorized them using a customized classification algorithm that is largely adapted from (Larsson, Nyström, Andersson, & Stridh, 2015; Larsson, Nyström, & Stridh, 2013). Here, we report the presence and durations of foveation types, as well as the main sequence of saccades across cognitive tasks. 

Methods

Subjects

Eye movements were recorded from two male Macaca mulatta monkeys (R and W) who had been trained on all three tasks. These monkeys were cared for per the Canadian Council for Animal Care Guidelines, and experiments were approved by the McGill University Animal Care Committee. Subject R completed 14 sessions and Subject W completed 52 sessions. 

Animals' preparation and set-ups

During a session, the animals were head-fixed and seated in custom primate chairs in front of a monitor (32° horizontally, 24° vertically) (Figure 1a). A video-oculography system (EyeLink 1000, SR Research, Ontario, Canada), sampling at 500 Hz, was positioned just below the monitor to track the left eye. The primate chair was fit with a two-axis joystick (M212, PQ Controls, Bristol, CT) used for free navigation in the VE tasks, allowing for forward and backward motion and right and left rotation turns. The computers used to run these experiments were an experimental control Mac (2 × quad-core 3.2 GHz Mac Pro with 8 GB of RAM and 512 MB of dedicated video memory) and the VE computer (eight core 3.4 GHz Windows 7 PC with 16 GB of RAM and 2 GB of dedicated video memory), the same setup as in Doucet, Gulli, and Martinez-Trujillo (2016). The VE was displayed on a 27″ ASUS VG278H monitor. The virtual environment was generated using the Unreal 3 engine, and is described in (Doucet et al., 2016) which also describes the open-source toolbox used for data acquisition and alignment. This toolbox allows the eye-tracker computer and the experiment control computer to be synchronized with submillisecond precision. Both computers were synchronized with the virtual environment computer with ±3 ms precision, well below the refresh rate of the screen (75 Hz), and within two samples of the EyeLink when sampling at 500 Hz. All samples were time-stamped for offline alignment. A reward system delivered a juice reward after correct trials. 

Figure 1

 

Overview of setup and tasks. (a) Experimental set-up: The monkey was head-fixed in a custom primate chair that had a two-axis joystick attached, positioned in from of the display screen. The EyeLink infrared camera and illuminator tracked the subject's eye. Juice reward was delivered via the sipper tube. (b) Cued Saccade task. A white dot was presented at one of nine locations on a gray screen, and the monkey had to saccade to the target and maintain fixation for 1–1.5 s to receive a reward. (c) Learning task: The monkey would navigate to the end of the corridor, and two out of three objects would appear. The reward associated with each object depended on the context of the trial, given by the texture on the walls. In this session, red was the highest reward in the Wood context, and Green was the highest reward in the Steel context; blue was worth the same amount in either context. (d) Foraging task: A red volume (virtual fog) would appear in one of 84 possible locations in the navigable environment inside the maze, and the monkey would have to navigate to the fog to receive a reward. (e) Virtual Environment: oblique view of the virtual environment used in the Learning task and the Foraging task.

Video-oculography was used as it is noninvasive, and has been shown to yield comparable accuracy to implanted eye-coils (Kimmel, Mammo, & Newsome, 2012; Van Der Geest & Frens, 2002). Issues that may still be present are an over estimation of the saccade velocity, and the postsaccadic oscillations, which are likely due to the pupil moving in relation to the eyeball (Nyström, Andersson, Magnusson, Pansell, & Hooge, 2015; Nyström, Hooge, & Holmqvist, 2013). However, these limitations do not affect the conclusions drawn here, since we are comparing data across different conditions using the same technique. 

Tasks

Cued Saccade task

A gray screen was presented, with matched luminance to the average luminance of the virtual environment. A white dot (0.5°) appeared at one of nine points on a grid (spaced at −12°, 0°, 12° horizontally, and −8°, 0° and 8° vertically), and the monkey had to fixate for 1000–1500 ms in order to receive a juice reward (Figure 1b). Intertrial intervals ranged from 2000–12000 ms; the subject's eyes were tracked for the duration, and trial start did not rely on the subject's eyes fixating a specific location. There were 20 ± 4 standard deviation (SD) trials completed prior to moving on to the VE tasks. Upon completion of the VE tasks, subjects again performed the Cued Saccade task, completing an additional 36 ± 21 SD trials. 

Learning task

The Learning task was embedded into a virtual environment that consisted of a long corridor branching out to two arms at each end (called the X-maze, Figure1e). Subjects were trained to use the joystick to freely navigate through the X-Maze while learning a context-object association (Figure 1c). On an individual trial, monkeys started in the Navigation period, and navigated from the North end to the South end, or vice versa. Upon entering the central corridor, one of two possible textures appeared on maze walls (context: wood or steel). When the monkey reached the branching point, a colored disc appeared in each of the two visible maze arms. Disc appearance was timestamped and used as the start of a Goals period, during which a decision had to be made and ended when the subjects navigated to the chosen disc to receive a reward. Subjects chose which disc to navigate to based on context-object associations learned within a single session, and received a juice reward for navigating towards the correct one. The three possible disc colors formed a juice reward hierarchy that was inverted between contexts (see Figure 1c, bottom right panel). Colors were pseudorandomly selected at session start and remained constant for the duration of the session. Concurrent with reward delivery, an auditory cue gave feedback on whether the selected object was the highest reward of the two available choices. On completed trials, the monkey simply kept navigating and turned around to travel to the other end of the maze to complete the next trial. Timed-out trials meant that the screen went black for at least 5 s of timeout, and then the camera jumped back to the middle of the maze to start a new trial. After removing interrupted or uncompleted trials, the subjects completed 238 ± 59 SD trials each day while virtual navigation training took approximately 5–7 days for each monkey. This includes only the time after the animal was chair-trained and had become familiar with the joystick. Basically, after the animals were exposed to the environment, they were navigating within 5–7 days. Training in the learning task to a proficiency where the rules could be learned quickly every day took 2–3 months. As the amount of reward was based on learning performance, and the task was difficult, the number of trials fluctuated significantly based on how motivated the monkey was and how quickly he learned the hierarchy. 

Foraging task

The Foraging task was embedded into the same X-Maze as the Learning task. In this task, there were no discs or context textures; rather, on each trial a semitransparent red column (Figure 1d) appeared in one of 84 equally spaced locations that covered the entire navigable arena of the virtual environment. Subjects navigated to the column's location to receive a juice reward, and then 200 ms after the reward had finished, the next red column would appear. The column was tall enough to be visible above the walls of the maze regardless of column or monkey position. Trial lengths were a maximum of 30 s, but were otherwise not controlled for. An average of 68 ± 26 trials were completed per session. 

Reward levels in the Cued Saccade task and Foraging task as well as the middle reward value of the Learning task hierarchy were all equal. A full session took from 90 to 120 min. There is some variability in the numbers of trials completed in each task because reward levels per trial are variable in the learning task, as were the daily learning rate and motivation. This meant that Foraging and Cued Saccade task trials might have to be added or cut to ensure that total reward levels were within a healthy range. 

Eye movement classification

Previous eye movement classification methods, such as pure velocity/acceleration thresholds or combinations with principal component analysis, poorly discriminate between smooth pursuits and saccades (Andersson, Larsson, Holmqvist, Stridh, & Nyström, 2016; Komogortsev & Gowda, 2010). For this reason, the current eye movement classification algorithms were iterative and data-driven (inspired by Larsson et al., 2015; Larsson et al., 2013). Many of the thresholds and a few of the calculations were adjusted to account for faster saccades and shorter foveations of monkeys (Berg, Boehnke, Marino, Munoz, & Itti, 2009), as well as to benefit from and for better signal to noise ratios as a result of head restraining. 

Prior to eye movement classification, data had to be cleaned and smoothed. First, blinks, periods of lost signal, off-screen eye positions, and corneal-loss spikes were identified as noise and removed from the eye position signal. Corneal-loss spikes are caused when the corneal reflection is lost and then regained, causing a spike in the signal. The EyeLink software uses a heuristic filtering algorithm (personal communication) outlined in Stampe (1993) that identifies spikes in the signal and replaces them with an adjacent value. While this is effective for most of the signal, we found that artefacts were still detectable if multiple corneal spikes occurred in succession. This was most likely caused by a drooping eyelid, and so would also interfere with gaze estimation, so we could not trust this data and had to remove it. Identification was done in a similar fashion to that outlined in Larsson et al. (2013). An iterative acceleration threshold identified high acceleration periods (outlined in the section on saccade identification). Any high acceleration period with a Euclidean distance between the onset and the offset of less than 0.3°, but with a one-sample displacement of more than 0.3°, is possibly a spike. To ensure it was not simply a postsaccadic oscillation (PSO), the average velocity of the period had to be less than the velocity of the preceding 10 ms. This check meant that spikes of more than 0.3° would be caught, but PSOs would not be identified, as the velocity of a PSO will always be less than the preceding saccade (Larsson et al., 2013). Any periods with a spike were removed. Next, the remaining valid signal was smoothed with a second-order Savitzky-Golay filter (as used in Nyström & Holmqvist, 2010) with a window of 11 samples. Finally, each transition from valid to noisy data and noisy to valid was analyzed to identify a stable signal (at least 6 ms with velocity consistently below 40°/s) adjacent to the noisy data. Periods of data between the stable signals were removed from further analysis, leaving only clean, preprocessed eye signal. 

Eye movement classification proceeded with the identification of saccadic intervals and the identification of the onset and offset points. The initial identification of saccades was done based on the high angular acceleration of the eye during a saccade. This was calculated separately for the x and y dimensions. The angular velocity of the smoothed eye signal was first calculated directly for each dimension as the difference between samples divided by the sampling rate. The angular acceleration was calculated by convolving the velocity with a filter, h(n), (Equation 1) that also smooths the signal,  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\bf{\alpha}}\)\(\def\bupbeta{\bf{\beta}}\)\(\def\bupgamma{\bf{\gamma}}\)\(\def\bupdelta{\bf{\delta}}\)\(\def\bupvarepsilon{\bf{\varepsilon}}\)\(\def\bupzeta{\bf{\zeta}}\)\(\def\bupeta{\bf{\eta}}\)\(\def\buptheta{\bf{\theta}}\)\(\def\bupiota{\bf{\iota}}\)\(\def\bupkappa{\bf{\kappa}}\)\(\def\buplambda{\bf{\lambda}}\)\(\def\bupmu{\bf{\mu}}\)\(\def\bupnu{\bf{\nu}}\)\(\def\bupxi{\bf{\xi}}\)\(\def\bupomicron{\bf{\micron}}\)\(\def\buppi{\bf{\pi}}\)\(\def\buprho{\bf{\rho}}\)\(\def\bupsigma{\bf{\sigma}}\)\(\def\buptau{\bf{\tau}}\)\(\def\bupupsilon{\bf{\upsilon}}\)\(\def\bupphi{\bf{\phi}}\)\(\def\bupchi{\bf{\chi}}\)\(\def\buppsy{\bf{\psy}}\)\(\def\bupomega{\bf{\omega}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\begin{equation}\tag{1}h\left( n \right) = 20\left( {\matrix{ { - 1}\hskip+4pt { - 1} \hskip+4pt { - 1} \hskip+4pt { - 1} \hskip+4pt 0 \hskip+4pt 1 \hskip+4pt 1 \hskip+4pt 1 \hskip+4pt 1 \cr } } \right)\end{equation}

and was adapted from the original in (Engbert & Kliegl, 2003). The thresholds for acceleration for the x and y dimensions were calculated separately using an iterative method adapted from (Nyström & Holmqvist, 2010). An initial threshold (T) of 10,000°/s² was set, and the mean (M) and standard deviation (SD) of all the values below T were calculated. An updated threshold (nT) was set at 6 SD above the mean (nT = M + 6 SD). The threshold T was then set at nT, and this calculation was repeated until the net change was less than 1°/s² per iteration. An index vector for each dimension was then generated to differentiate the points that are above the respective thresholds. Index vectors were then combined to identify when the eye was in a high acceleration period, representing potential saccadic periods. To correctly identify saccades and to avoid noise, we relied on the threshold for minimum intersaccadic intervals of 40 ms and minimum saccade duration of 10 ms used by Larsson et al. (2013) and Nyström and Holmqvist (2010). Specifically, potential saccadic periods less than 40 ms apart were combined, while all periods lasting less than 10 ms were ignored. This generated potential saccadic periods from which precise onset and offset points could be calculated. It is important to note that whereas intersaccadic intervals of less than 40 ms might be possible, we found that only 0.1% of these intervals were between 40–50 ms, whereas 1.5% were between 90–100 ms.  

Saccade onset and offset were calculated for each putative saccade period based on two criteria that rely on the consistency of direction, as well as a velocity criterion. The first is the main direction, as there is very little deviation in direction over the course of the movement once the saccade has started. We first determined the instantaneous velocity and direction for each sample over the saccadic period to identify the peak velocity and main direction. The main direction was obtained from the mean direction at the time of peak velocity and its two adjacent samples. Moving away from the point of peak velocity in either direction in time, two velocity thresholds were set: a continuous change of > 20° for at least three samples, or an acute change of > 60° at one sample. Any threshold crossing events occurring before the peak velocity could indicate a potential saccade onset, and later, a potential saccade offset. We further tested the sample to sample changes in direction, using the same thresholds (> 20° continuous, and > 60° acute), and moving backwards in time from the peak velocity to find the onset, and forwards to find the offset. For these two criteria, the threshold crossing closest to the peak velocity point was used in the next step. The final criterion was that the velocity at an onset/offset must be below the greater of 1/5 the peak velocity or 30°/s. If the direction threshold was crossed before the velocity had fallen below the velocity threshold, the onset/offset was moved to the first point below the threshold. These direction thresholds were modified from the original because the stability of the signal and the velocity of the saccades meant that during the saccade the direction was very stable. The original thresholds from Larsson et al. (2013) had a very hard time selecting accurate endpoints during catch-up saccades, where smooth pursuit and saccades were in the same direction. 

Intersaccadic intervals contained three possible types of eye movements: postsaccadic oscillations, smooth pursuits, and fixations. Postsaccadic oscillations (PSOs) are a stereotypic signal observed at the end of some saccades, and are found when eye position is measured with both scleral coils and video-oculography (Kimmel et al., 2012). In video-oculography, this movement is partially driven by the wobbling of the pupil in relation to the iris at the end of saccades (Nyström et al., 2015). While PSOs are not a behavior per se, the following differentiation of smooth pursuit and fixation are facilitated by separately classifying them and removing them from the signal. The identification method was done in the x and y dimensions separately, by analyzing 40 ms (20 samples) after every saccade. Starting at the last of the 20 samples, we looked for the point where the linear signal turned into an oscillation. To do this, we looked for a change in velocity starting with the third last point fitting a straight line to all the points to the end, and another straight line only with the point before it, and compared the slopes (velocity). If the difference in slopes, or acceleration, was less than 17°/s², (a value in the 95th percentile of foveation accelerations in VE tasks) the test point was moved one index towards the start and the test was repeated. Once an inflection point was found, the eye position at the inflection point was subtracted from the signal, transposing the inflection to zero, and the signal to the right of the inflection was set to zero. As the oscillations now resembled a decaying impulse, the “prony” MATLAB function was used to calculate an all pole filter with one to four poles (coefficients), and then the impulse response was calculated for each of these filters using the MATLAB function “impz.” The root mean squared error (RMSE) was calculated between the data and the four impulse responses, and normalized by the maximum absolute value of the signal. The higher order coefficient models only replaced the first order model if the RMSE was at least 5% lower. If the RMSE of none of the models was below 0.15, then the first point of the signal was dropped, and the signal remodeled. This was repeated until there was an RMSE below 0.15. Once a model was selected, the decay was calculated, and the offset of the PSO was set when the absolute value of the signal was less than the decaying signal plus 0.08 for at least three samples. At this point, the oscillating signal had decayed such that fluctuations would be so small that they would not likely interfere with foveation discrimination. Having modeled the signal, there are three checks for stereotypical PSO: The largest pole coefficient must be below 0.89, which sets a limit for the speed of the decay, where higher pole coefficients have slower decay. The minimum amplitude of the oscillation was set at 0.15° to only include oscillations large enough to alter foveation classification. Finally, as a further check to prevent the inclusion of slow movements, a ratio of the total range of the signal divided by the duration was calculated, and had to be above 15°/s. If these criteria aren't met, then the signal is not categorized as a PSO. 

Differentiation between foveation types (fixations and smooth pursuits) relies on sample directions and ratios of distances rather than velocities (Larsson et al., 2015). Though both foveation types keep an object centered on the fovea, this distinction is important when studying eye movement behavior, kinematics, and mechanisms, since the neural circuits that enable these behaviors are very different (Büttner-Ennever & Horn, 1997). First, the yet unclassified eye signal is tested for the presence or absence of a consistent mean direction in 22 ms bins using a Rayleigh test on the sample-to-sample direction within that window (p < 0.01). The p value for the bin was assigned to the index for each constituent point. Because of the overlap of the bins, for points that were part of overlaps, the average p value of the two bins was assigned. The signal was then analyzed for longer sections of at least 40 ms that had a p value consistently above or below 0.01. For each section, four measures were calculated, each with a threshold to define smooth pursuit: (a) Dispersion of samples is the ratio of the second principal component divided by the first principal component, with the threshold at < 0.45. Lower dispersion suggests that the movement is linear. (b) Consistency of direction is the Euclidean distance between the first and last points of the section divided by the first principal component, with a threshold > 0.5. A number close to one means that the variability in the eye's movement is largely described by the movement from the initial position to the final position. (c) Total path displacement ratio is the Euclidean distance between the first and last samples divided by the sum of the trajectory, and a threshold > 0.3. While noise or nonlinear movement will increase the trajectory length, a ratio closer to 1 means it is more linear. And (d) The total spatial range of the section is the diagonal of the rectangle with dimensions of x range and y range of the samples (this inscribed all the points in the section), and had a threshold > 1.5°. This is effectively the maximum range of a fixation, and any foveation that is travelling more than this is likely a smooth pursuit. 

If all values crossed the threshold, then the section was classified as smooth pursuit, and if all the values fell on the fixation side, then it was classified as a fixation. Sections that did not uniformly meet smooth pursuit or fixation thresholds, or were not long enough to be analyzed, were added together with other sections of the foveation with a mean direction < 45° different. The path displacement is recalculated, and if it crosses the smooth pursuit threshold, then it is defined as a smooth pursuit, and if it does not, then the new spatial range is calculated. If the spatial range of the sample exceeds the minimum range for a smooth pursuit (1.0°), then it is classified as a smooth pursuit, otherwise as a fixation. Whereas the first two thresholds are the same as in Larsson et al. (2015), the rest are slightly modified to account for the shorter foveation durations of macaques (Berg et al., 2009). A smooth pursuit for a shorter duration would travel a shorter distance, and a shorter fixation would have less drift, which allowed us to lower these thresholds. 

Whereas other studies have carried out systematic comparisons of classification algorithms over numerous subjects, as well as against expert classifications, this method was shown by Larsson et al. (2015) to be highly efficient. 

Characterization of eye movements

Before analyzing the different behaviors, we analyzed the proportion of trial time where the gaze was on the screen. This excluded periods when the monkey was blinking, closed its eyes, or looked away from the screen. We compared the metric across tasks, and compared proportions during the two periods of the Learning task. The Navigation period started 1 s after reward was finished and ended when the subject reached the Goals area (Figure 1c). The Goals period started upon entry into the Goals area, when the two discs appeared, and finished when a disc had been navigated to and a reward was given. Having classified the eye signal into distinct categories, we ran an ANOVA to compare the distinct categories as a proportion of the total time looking at the screen. Populations were taken as the proportions for each experimental session. Posthoc tests were Bonferroni corrected for multiple comparisons. 

We classified foveations into three types: pure fixations, pure smooth pursuits, and combinations. Combinations were foveations that contained at least one section that was classified as fixation and at least one section that was classified as smooth pursuit. For each task, we calculated the percentages of the total foveations classified as the three types. For comparisons, we ran ANOVAs for the effect of task on the classification percentage, and ran posthoc tests with Bonferroni corrections for multiple comparisons comparing fixations and smooth pursuit percentages within tasks. Because combinations were a small proportion of all foveations, they were not considered in future analyses. 

We used a Kolmogorov-Smirnov test to compare the durations between fixations and smooth pursuits, only including foveations that occurred between two classified saccades. Foveations before or after removed data, such as those interrupted by blinks, were not used for this analysis, because the start or end times were unreliable. 

Main sequence analysis

For the analysis of the main sequence, we separated saccades into bins of 3° of amplitude, starting at 2°. A main sequence was generated by taking the medians for each bin and compared each condition using a repeated measures ANOVA with bins treated as the subject and task as the treatment. We initially generated main sequences for the three tasks by using every saccade. This method did not control for the kinematics of the saccades, so we also ran a saccade matching algorithm to account for the influence of saccade start location and direction before recalculating the ANOVA. Using the same bins, we found saccades from that session that matched the direction with a tolerance of ±10°, and the starting location within 5°. Strict saccade matching meant that only 10%–30% of saccades could be matched. We combined our matched saccades across sessions for our analyses. Results were consistent in both monkeys separately, but are presented here as pooled. 

The second analysis used model fitting to get a measurement of the effect of the tested condition on the peak velocity. The main sequence for peak velocity follows a steep initial curve that levels out as it reaches a maximum peak velocity, and can be fit to an exponential equation seen in Equation 2 (Baloh, Sills, Kumley, & Honrubia, 1975). As there are two variables in this equation, comparing the confidence intervals of two separately calculated fits can lead to results that are difficult to interpret. To address this problem, we instead used a categorical variable to calculate a nonlinear fit using MATLAB's fitnlm to find the difference between the maximum velocities of the two fits, and to get a statistical test for the significance of the value. The benefit of this analysis is that it takes all the saccades into account, rather than the means or medians of a bin. This method can be used to calculate a scalar multiplication relationship between the classes and does not require a bin size.  

\begin{equation}\tag{2}PV = M \times \left( {1 - {e^{\left( { - {A \over S}} \right)}}} \right)\end{equation}

 

\begin{equation}\tag{3}PV = \left( {M + K \times B} \right) \times \left( {1 - {e^{\left( { - {A \over S}} \right)}}} \right)\end{equation}

 

\begin{equation}\tag{4}PV = C \times M \times \left( {1 - {e^{\left( { - {A \over S}} \right)}}} \right),C = {{M + K \times B} \over M}\end{equation}

 

In this model, PV is the peak velocity for the saccade, A is the amplitude of the saccade, and K is the binary categorical marker for the condition (either 1 or 0). The fitted parameters, M and S are the maximum velocity that the logarithmic equation approaches and the slope, respectively. B is the scalar modifier that describes the difference of the peak velocity between the conditions. For models that have more than two different populations, there is a B for each population except the first. Equations 3 and 4 are equivalent, with Equation 4 highlighting the transformation required to interpret the categorical addition (B) as a scalar multiplication (C). C is used as a comparison in our results so that the effect can be compared across separate analyses without having to match saccades across four or five different conditions, and C is presented alongside the fits in Figures 4 through 6. From fitnlm, we can get the estimates and p value for each of the parameters for significance testing. 

Figure 2

 

Eye position and behavior classification examples and proportions. (a) Overhead plot of subject position during an example Learning task trial with color mapping on to the time color bar in (c). (b) Example trial eye position plotted in screen coordinates. Each dot is a smoothed sample, and its color indicates the classification: green—saccade, red—fixation, blue—smooth pursuit and magenta—postsaccadic oscillation (PSO). (c) Example eye position, split into x and y components. The background denotes the period of the task. Orange background is the Navigation period, and Green background is the Goals period. (d) Quantification of the change in gaze on screen behavior across the tasks and task periods. Whereas there is less gaze on screen behavior in the Learning task, analyzing the two periods separately show that this is driven by differences in the two periods, with high gaze on screen behavior when there are rewarded targets on the screen.

Figure 3

 

Proportions and durations of eye movements across tasks. (a) The proportion of onscreen time spent on each behavior changes between tasks. Error bars represent SD. (b) Proportions of total foveations by number. There are mostly fixations in the Cued task, with smooth pursuits and combination foveations being approximately equal. There are few combination movements in the VE tasks with approximately equal numbers of smooth pursuits and fixations. (c) Population density plots for foveation duration in the VE tasks. The cumulative distribution function plots are inset. The medians are indicated by circles (Foraging task) and stars (Learning task) in red for fixations and blue for smooth pursuits.

Figure 4

 

Comparisons of saccadic main sequences across the three tasks showing faster saccades in VE, with matched saccades showing an effect of task demands within VE. (a) Saccade peak velocity as a function of saccade amplitude for all saccades in each of the tasks. (b) Nonlinear model fit with significant modifiers for both VE. The equation is indicated. C is equal to the scalar modifier for the task. (c, d) Same plots as (a, b), but for matched saccades. The categorical modifier for the linear model for Foraging was more than twice that of the modifier for Learning. Error bars in (a) and (c) represent SD, and dashed lines in (b) and (d) represent 95% CI.

Figure 5

 

Changes in VE main sequence are a result of task demands, not static/dynamic changes. (a) Saccade peak velocity as a function of saccade amplitude during Static versus Dynamic stimuli in Goals period of Learning task; error bars represent SD. (b) Nonlinear model fit of static and dynamic saccades. The equation is indicated. C is equal to the scalar modifier for the task; dashed lines represent simultaneous 95% CI for the fit. (c, d) Same as (a, b), but comparing saccades in the Navigating and Goals periods in the Learning task. Nonlinear model fit with a significant modifier for Goals period saccades. (e, f) Comparison of Goals period and Foraging saccades, same layout as (a, b).

Figure 6

 

While discreet visual targets elicit faster saccades than to blank regions or to VE, saccades during exploratory VE navigation are still faster than to blank screen. (a) Saccade peak velocity as a function of saccade amplitude for visually guided saccades (VGS) and nonvisually guided saccades (NVGS) during the Cued Saccade task. Error bars represent SD. (b) Nonlinear model for VGS and NVGS main sequence; dashed lines represent 95% confidence intervals. (c, d) Similar plots as (a, b), but comparing VGS from the Cued Saccade task and saccades from the Goals period of the learning task. (e, f) Same as above plots, comparing NVGS saccades from Cued Saccade task and the Navigation period of the Learning task.

After analyzing the main sequence across the three tasks, we compared the main sequences of the static and dynamic phases of VE navigation. To control for possible effects of task demands, we only compared saccades during the Goals period of the Learning task. Static phase saccades had to occur within a period where all the frames had the same orientation and location. Dynamic phase saccades had to occur over frames during navigation. 

We also wanted to analyze the effect of task demands in the Learning task by comparing saccades in the two task periods. We had observed that there were differences in screen watching during these periods, and so we compared the proportions of the periods where the subject's gaze was on the screen. These were not normally distributed and so were compared using a Wilcoxon rank sum test. Having found a difference, we compared saccade velocities between the two periods. Navigation period saccades were saccades that occurred from 1 s after reward offset to just before the goals appeared, and the Goals period was from goal appearance to reward onset. Goals period saccades were also compared to Foraging task saccades. 

We compared the saccades in the Cued saccade task by splitting them up into saccades with an offset within 1° of the perimeter of the visible target (visually guided saccades, VGS, and all other saccades to regions of the screen (nonvisually guided saccades NVGS). Other analyses included comparing matched VGS saccades to Goals period saccades and Foraging task saccades separately, and the NVGS saccades to the Navigation period saccades. 

Results

Monkeys performed three tasks each day, with eye movement classification and characterization done in each task separately (Figure 2). We sought to compare eye movement proportion and kinematics across each task. Table 1 outlines some basic characteristics about the three tasks, trial lengths, gaze on screen time, and numbers of saccades, fixations, smooth pursuits, and combination foveations that had both smooth pursuit and fixation characteristics. 

Table 1

 

Summary statistics for trials of the three different tasks, as well as the separate periods of the Learning task. Notes: Reported values are median and interquartile range (IQR) as many of these measures were not normally distributed.

 

The first behavior that impacts our data collection is when the monkey is looking at the screen and not blinking, looking off screen, or looking at the juicer that was positioned to not interfere with screen viewing. We first compared the proportion of trial time with gaze on screen behavior between the tasks and the periods of the Learning task (Figure 2d). There was a significant effect of task on the gaze on screen behavior using a repeated measures ANOVA: F(2, 130) = 211.9, p ≪ 0.05. To test what the differences that were driving this effect, we found there was no difference between the Cued saccade task, mean = 78.5%, (SD) = 11.0, and the Foraging task, mean = 78.2%, SD = 10.6, paired t-test, t(65) = 0.83, p = 0.8; whereas the gaze on screen was lower in the Learning task, mean = 56.2%, SD = 8.23, when compared to the Cued saccade task, paired t-test, t(65) = 14.78, p ≪ 0.01, and the foraging task, paired t-test, t(65) = 20.37, p ≪ 0.01, all comparisons Bonferroni corrected for multiple comparisons. We tested the periods of the Learning task and found that the Goals period had a much higher percent of trial time with gaze on screen, mean = 90.7%, SD = 4.6, than the Navigation period, mean = 47.2%, SD = 9.4, paired t-test, t(65) = 54.6, p ≪ 0.01. 

We used an ANOVA to determine whether the proportion of time spent performing each eye movement type varied across tasks (Figure 3a). There was a significant effect of task for the proportion of time classified for each movement type: fixation, F(2, 195) = 656, p ≪ 0.01; smooth pursuit, F(2, 195) = 400, p ≪ 0.01; saccades, F(2, 195) = 402, p ≪ 0.01. The proportion of time classified as either foveation type changed dramatically across tasks; time spent fixating was significantly higher during the Cued Saccade task, 81.6% ± 1.3 SEM than the Foraging task, 34.2% ± 1.0 SEM, two-way t-test, t(130) = 27.1, p ≪ 0.01, and the Learning task, 35.9% ± 0.4 SEM, two-way t-test, t(130) = 31.6, p ≪ 0.01. Fixations were not significantly different between the VE tasks, Foraging and Learning, two-way t-test, t(130) = 1.9, p = 0.06. The proportion of time classified as smooth pursuits was significantly lower in the Cued Saccade task, 13.3% ± 1.0 SEM, than in the Foraging task, 43.3% ± 1.0 SEM, two-way t-test, t(130) = 17.6, p < 0.01 and the Learning task, 51.9% ± 0.5 SEM, two-way t-test, t(130) = 32.4, p < 0.01. Smooth pursuit is mostly a misclassification in the Cued Saccade task, as there are no moving stimuli on the screen, and this is addressed further in the Discussion. The proportion of time classified as smooth pursuits across VE tasks was also significantly different, paired t-test, (130) = 7.0, p < 0.01. 

In addition to comparing the proportion of time spent on each eye movement type, we also compared the numbers of pure fixations and pure smooth pursuits across tasks within each session. Again, there was a significant effect of task on both fixations, F(2, 195) = 659, p ≪ 0.01, and smooth pursuits, F(2, 195) = 1007, p ≪ 0.01. Table 2, row 7 shows a ratio of 7:1 between the number of fixations and smooth pursuits in the Cued Saccade task, and a ratio of 4:5 (rows 8 and 9) in the Foraging and Learning tasks. In each task, less than 15% of all foveations contained segments of both smooth pursuit and fixation. Misclassifications caused by drift or vergence are covered in the Discussion. 

Table 2

 

Posthoc tests for effect of task on proportion of total foveations where fixation or smooth pursuit are expressed, and tests of differences in proportion within a task. Notes: * indicates significant difference with an α of 0.05 Bonferroni corrected for multiple comparisons.

 

We next compared the durations of all foveations in the Foraging task, median = 198 ms, interquartile range (IQR) = 92 ms, and the Learning task, median = 202 ms, IQR = 98 ms. The difference in medians was 4 ms in the Kolmogorov-Smirnov D(83412, 285763) = 0.032, p ≪ 0.01. We then compared the populations of fixation and smooth pursuits durations within each task (Figure 3b). There was only a small difference between foveation types for the Foraging task: fixation median = 182 ms, IQR = 92 ms; smooth pursuit median = 188 ms, IQR = 92 ms, D(33979, 45222) = 0.068 p ≪ 0.01. There was a slightly larger difference in the Learning task: fixation median = 176 ms, IQR = 80 ms; smooth pursuit median = 216 ms, IQR = 116 ms, D(128468, 127010) = 0.236, p ≪ 0.01. We did not carry out the comparisons of foveation duration in the Cued Saccade task because this task required the subject to maintain fixation for extended durations to successfully complete a trial. 

We did, however, compare the foveation durations between the Navigation period (median = 206 ms, IQR = 98 ms) and Goals period (median = 180 ms IQR = 102ms) of the Learning task and found a significant difference in Kolmogorov-Smirnov D(141980, 116028) = 0.17, p ≪ 0.01. This can be explained by a change in behavior, as during the Goals period the animals needed to identify the target features to exploit the reward; therefore, they tried to sample each object at a high frequency, reducing each foveation duration. On the other hand, during navigation the animal simply explored the objects in the environment, without having particular objects competing for attention based on task demands. 

Main sequence analyses

Next, we analyzed the main sequence of saccades in the virtual environment tasks and the Cued Saccade task. We initially compared all saccades from the three tasks (Figure 4a and b). There was a main effect of task on the repeated measures ANOVA, F(2, 16) = 28.61, p ≪ 0.01. Table 3 shows the posthoc tests which indicate that the effect of task is driven by lower peak velocities in the Cued saccade task. The nonlinear model velocity, F(4, 530252) = 3740000, p ≪ 0.01, R²adjusted = 0.847, had categorical modifiers that were significant for the VE tasks compared the Cued Saccade task: Foraging B = 160.5, t(530252) = 104.7, p ≪ 0.01; Learning B = 153.4, t(530252) = 119.79, p ≪ 0.01. When we reanalyzed the main sequence with matched saccades to control for saccade start location and direction (Figure 4c and d), we still had a significant effect of task, F(2, 16) = 19.7, p ≪ 0.01. However, as the posthoc analyses in Table 3 show, when saccades are matched, there are significant differences in peak velocities for bins between each of the tasks. The model was also of a similar fit, F(4, 23894) = 1370000, p ≪ 0.01, R²adjusted = 0.74, and the categorical modifier for Foraging, B = 116.2, t(23894) = 32.1, p ≪ 0.01, was almost twice as high as for the Learning task, B = 52.8, t(23894) = 14.7, p ≪ 0.01. Further tests were carried out to analyze what could be driving these changes in peak velocity. 

Table 3

 

Posthoc t tests for the peak velocities in the main sequence between tasks for nonmatched and matched saccades. Notes: * indicates significant difference with an α of 0.05 Bonferroni corrected for multiple comparisons.

 

We tested whether static or dynamic periods (stationary or navigating phases) could have caused this difference within the VE tasks. To control for task demands, we only compared saccades in the Learning task where the rewarded objects were visible (Figure 5a and b). Using a repeated measures ANOVA, we found that there was no difference in the main sequences of saccades during static phases compared to during dynamic phases, F(1, 8) = 1.89, p = 0.207. There was a significant categorical nonlinear fit, F(3, 1753) = 36100, p ≪ 0.01, R²adjusted = 0.894 with a nonsignificant categorical difference modifier for the dynamic phase, B = 4.37, t(1753) = 0.585, p = 0.559. 

We tested the different periods of the Learning task to analyze the effect of task demands (Figure 5c and d), comparing when the monkey was just in a Navigating period with when the monkey had to decide between the two objects in the Goals period. We suspected there might be a difference between these two periods because the monkey had its gaze on the screen for a greater proportion of the time during the Goals period (median = 0.91, IQR = 0.05) than the Navigation period (median = 0.46, IQR = 0.13), which was significant on a Wilcoxon rank-sum test p ≪ 0.01. When testing the main sequence, there was a significant difference in the repeated measures ANOVA, F(1, 8) = 9.26, p = 0.021, with faster saccades during the Goals period. We found a significant nonlinear model, F(3, 111203) = 10,600,000, p ≪ 0.01, R²adjusted = 0.816, with a significant categorical modifier for Goals, B = 60.37 t(111203) = 43.676, p ≪ 0.01. This translates into a gain of 1.05 ± 0.002. 

Having found an effect of task demands, we compared the saccades of just the Goals period of the Learning task and the Foraging task (Figure 5e and f). In these two periods, the monkey navigates directly towards a rewarded object. Under these conditions, we found no significant effect in our ANOVA, F(1, 8) = 3.1, p = 0.12. The nonlinear model was significant, F(3, 73127) = 941000, p ≪ 0.01, R²adjusted = 0.877, with a nonsignificant categorical modifier for Goals, B = −2.11 t(73127) = −1.36, p = 0.175. 

To better analyze the differences between the Cued Saccade task and the VE tasks, we first compared the main sequences for visually guided saccades (VGS) and nonvisually guided saccades (NVGS). When comparing these, it is important to note that the ratio of VGS to NVGS had a mean of 0.28 ± 0.06 SD. We replicated previous findings from (Edelman, Valenzuela, & Barton, 2006; see Figure 6a and b) that showed there was a difference between these groups of saccades with our ANOVA, F(1, 7) = 22.8, p ≪ 0.01. The model had a significant positive modifier for NVGS, Beta = 205.9, t(1663) =18.7, p ≪ 0.01, that equated to a gain of 1.18 ± 0.02. 

To compare the saccades in the VE to saccades during classical tasks, we compared the saccades during the Goals period to the VGS in the Cued Saccade task (Figure 6c and d). The VGS were significantly faster than the Goals period saccades, F(1, 6) = 10.12, p = 0.019. The nonlinear model was significant, F(3, 1921) = 39,500, p ≪ 0.01, R²adjusted = 0.923 with a significant negative categorical modifier for Goals, Beta = −107.1, t(1921) = −12.23, p ≪ 0.01. This result equaled a gain of 0.93 ± 0.01. Results were similar when comparing Foraging task saccades to VGS with a gain of 0.92 ± 0.01. 

We also compared the Navigation saccades to NVGS in the Cued Saccade task (Figure 6e and f). Navigation period saccades were significantly faster as measured by the ANOVA, F(1, 8) = 35.58, p ≪ 0.01. The nonlinear model, F(3, 25133) =171,000, p ≪ 0.01, R²adjusted = 0.661, had a significant categorical modifier, Beta = 71.8, t(25133) = 25.5, p ≪ 0.01. This translated to a significant gain for the Navigation period over the NVGS of 1.07 ± 0.01. 

Whereas a thorough comparison of the effect of all categories would require finding saccades that matched across all the categories, this would invalidate too many saccades to be a viable analysis for our data set. However, comparing the C values of the fits to a constant condition can give an approximate difference and make condition comparisons easier. To generate the summary figure presented in Figure 7, we chose the slowest population of saccades (NVGS), and plotted the C values for the other conditions. While C values for the conditions that were compared against the NVGS condition are readily available, C values for the Goals and Foraging tasks had to be derived in the following manner: Because there was a C value for VGS versus NVGS (1.18), and a C value for Goals versus VGS (0.97), we could multiply the Goals versus VGS C value by the VGS versus NVGS value to get an approximate Goals versus NVGS C. As we could also do this using the Goals versus Navigation C value, we took the average of the two derived values. As there was no significant C value for the Static versus Dynamic phases of the Goals periods, we plotted both in the summary to include this effect. We used the same method to calculate the C value for the Foraging task. 

Figure 7

 

Summary of multiplicative gain (C value) in each condition relative to Nonvisually guided saccades (NVGS). The C value for VGS (blue) and Navigation (orange) saccades compared with the NVGS condition are directly from the models. C values for the other conditions, Goals period static and dynamic (green) and Foraging (red) were approximated and are marked with asterisks.

Discussion

Proportion of eye foveation types across tasks

Our experiment aimed at characterizing oculomotor behavior of nonhuman primates in virtual environment tasks. As well as looking at saccade behavior, we wanted to look at fixation and smooth pursuit behaviors, and so we had to find a method for classifying these movements. We could not find studies in NHPs in VEs that distinguished between types of foveations and reported proportions of each, and so we looked to human research. One strategy that has been used by Hayhoe and coworkers is to either consider all eye signal near an object regardless of velocity a foveation (Hamid, Stankiewicz, & Hayhoe, 2010), or all eye signal below a calculated velocity threshold for a minimum duration as an information gathering foveation (Kit et al., 2014). Another method that did distinguish between smooth pursuits and fixations was used for a head-mounted display virtual reality task; however, this was for a prediction task, and not a navigation task (Diaz, Cooper, Kit, & Hayhoe, 2013). The reported data from this experiment was for following an object, and not exploration. The method was also for a lower sampling rate, and so we didn't use this method so that we could take advantage of our higher temporal resolution. Relying on methods developed by Larsson and colleagues (2013, 2015) for dynamic stimuli, we found that with some modifications the algorithms worked quite well for our NHP data. The code for these algorithms is available on the lab website (http://martinezlab.robarts.ca). 

Our first set of analyses focused on the foveations, specifically the classifications of smooth pursuits and fixations. We found that in terms of proportions of time and numbers of movements within a task, fixations to smooth pursuits ratios were higher in the Cued Saccade task than in the VE tasks. There is no real standard against which to compare these results, as we could not find these proportions reported in NHPs for movies or virtual navigation. 

In the VE tasks, the proportion of time spent in smooth pursuit was slightly higher than that of fixations. The ratio of number of smooth pursuits to fixations was 5:4, so there were also more sheer number of smooth pursuit movements, and this wasn't simply because the smooth pursuit movements were longer. In the VE tasks the dynamic stimuli generated by virtual navigation induced smooth pursuit foveations very often when compared to the Cued saccade task and likely when compared to any other task that uses static displays (e.g., free viewing pictures). This result highlights the importance of smooth pursuit eye movements when interacting with real dynamic environments. This is a common situation when considering that nonhuman primates and humans create these dynamic retinal patterns during locomotion, or when moving the head. Something to consider is that in many visual experiments in nonhuman primates the head remains fixed and displays are static (but see also Martinez-Trujillo, Medendorp, Wang, & Crawford, 2004). This may constrain the generality of the results when considering the variables we have isolated in this study (e.g., foveation types or estimated duration). 

In the static Cued Saccade task, our algorithm classified a mean of 13% and a median of 10% of the onscreen time as smooth pursuits, where there were no moving stimuli. In terms of proportion of total foveations, it classified a mean proportion of 0.09. We consider these likely to be misclassifications as there is nothing on the screen to visually pursue. In Larsson et al. (2015) their algorithm also classified 0.09 of the foveations as smooth pursuits in a static free viewing task. Whereas they speculate that many of these misclassifications are due to vergence movements, we have an added confounder where the foveations required in the Cued Saccade task were unnaturally long, and would have allowed drift to also contribute to misclassifications of fixations as smooth pursuits. It is certainly the case that we may have similar errors in the VE tasks; however, having ∼10% of fixation behavior being classified as smooth pursuit would not radically alter the ratios in the VE tasks. 

The increase in the proportion of smooth-pursuit fixations to saccade fixations in the VR tasks relative to the Cue Saccade task found in our study was predictable due to the presence of dynamic displays in the former but not the latter scenario. However, the fact that smooth pursuit foveations are equally used as fixations provides important information regarding how animals use one or the other type of foveation to explore the environment when navigating. Moreover, our animals were head fixed, so no VOR was present; it is possible that if the animals were head free, these proportions would change and smooth pursuit-VOR foveations might increase relative to fixations. An important aspect to consider is that during VR navigation, as well as during real navigation, the animal controls their trajectory and moves “at will.” This may be different from presenting a movie to the animals and recording the pattern of eye movements. Finally, our results may have some implications for physiological studies in monkeys exploring the interaction between eye movement signals and saccade-related signals such as efference copy (Sommer & Wurtz, 2002) and saccadic suppression (Thiele, 2002) since saccades and smooth pursuit are related to different temporal dynamics of activity profiles, for example, in Superior Colliculus neurons (Krauzlis, 2003). 

Foveation durations in the VE task

Despite the potential overestimation of the number of smooth pursuits, it is evident that the foveating behavior is split approximately evenly between fixations and smooth pursuits. If the lowered visual acuity of smooth pursuit movements described by Schütz et al. (2009) affects information acquisition in navigating behavior, then this is important to consider. The current task is not letter discrimination, and this processing could be done by peripheral vision, but as the eye was unconstrained, the monkeys could foveate both objects during the Goals period, and would switch between them. It is unlikely they were using peripheral vision to assess goal color, and instead were foveating on what they wanted to analyze. Keeping this in mind, we investigated whether there is a difference in the durations of the two types of foveations. If smooth pursuits require more time to acquire the relevant information, then the durations should be significantly longer. When we compared the foveations in the Foraging task, the Kolmogorov-Smirnov D showed that there was only a maximum of 3% difference between the cumulative density functions. There was also a 3% difference for the median duration. From the comparisons in this task, it is questionable as to whether there is a meaningful difference between duration required to acquire information during a smooth pursuit or a fixation. Further studies may discriminate whether this is idiosyncratic to our task or may be a more general principle that applies to many tasks. 

In the Learning task the median fixation duration was 40 ms shorter than the median smooth pursuit duration. In the Foraging task, median fixation duration was 6 ms shorter than smooth pursuits. The difference between medians was over five times as long in the Learning task as the foraging task, and the Kolmogorov-Smirnov D statistic was also about four times as large. In the Goals period of the Learning task, the animals had to divide their attention between two targets, at least at the point where they first perceive them, which was not the case in the Foraging task. It has been shown that dividing attention during smooth pursuit behavior affects performance (Niebergall, Huang, & Martinez-Trujillo, 2010), which in turn may affect the dynamic of eye movements. Future studies could control for variables such as location of the targets, number of targets, as well as computing optic flow at goal locations. 

There has not yet been a body of data on the effect of virtual navigation on nonhuman primate eye movements to compare our measurements with. Berg et al. (2009) recorded foveations during passively watching movies. They measured a median intersaccadic-interval (foveation) duration of ∼220 ms in nonhuman primates, which is 10% longer than in our tasks (median durations of ∼200 ms). This could be because of the change in engagement between watching a movie and actively navigating. In our Learning task, we found that there was a difference in median foveation duration between task periods. During the Navigation period, when the monkey was generally looking at the screen less, foveations were slightly longer. Foveations during the Goals period, in which the monkey had to gather information about two separate objects, were shorter. The monkey was rapidly sampling the objects and the environment and was also likely more engaged during this period. This suggests that not only the display dynamics play a role in fixation durations but also the demands of the task. 

Main sequence

A property of saccades that we analyzed was the main sequence, the relationship between a saccade's amplitude and its peak velocity and duration. The main sequence has been well characterized in both humans and nonhuman primates (Bahill et al., 1975; Edelman & Goldberg, 2003; Quaia, Paré, Wurtz, & Optican, 2000). There have been numerous studies that have identified different factors that can affect the main sequence, including arousal (Di Stasi, Catena et al., 2013; Galley, 1989) and intrinsic target value (Xu-Wilson, Zee, & Shadmehr, 2009). These are two variables that we predicted might change between a VE and the cued saccade task. When we initially compared the main sequences of all saccades during the three different tasks, we found a significant effect of task on the peak velocity of saccades. However, this analysis was not properly controlled, as the starting location and direction of the saccade can influence the velocity of the saccade (Becker, 1989; van Beers, 2007). To control for starting location and direction, we matched the saccades and reanalyzed the data, finding that while the Cued task still had the slowest saccades, now there was also a difference between the VE tasks. There are two specific things that were not controlled for in that initial matching of location and direction: presence of optic flow and level of engagement. The monkey could stop navigating during the task, eliminating the dynamic nature of the visual stimulus, and if that caused the change between the Cued Saccade task and the VE tasks, it might also cause the difference between the VE tasks. The speed of the optic flow was not controlled for, as it varied depending on where the eyes were on the screen; however, as the joystick allowed only one speed, navigating speed was controlled for. The difference in cognitive demands or engagement in the two periods of the Learning task could also be driving the change in the main sequence. We ran two tests to analyze what might be affecting the main sequence. 

Our initial comparison of static or dynamic phases was done just during the Goals period of the Learning task. There was no significant difference in velocities using the ANOVA test, and the beta for Category in the linear regression was not significant. Other studies using single moving targets found that the movement of a target could modify the main sequence, but that there are direction specific effects (de Brouwer, Missal, Barnes, & Lefèvre, 2002; Guan, Eggert, Bayer, & Büttner, 2005). De Brouwer et al. (2002) found that for saccades preceded by smooth pursuits, there was an interaction of the vectors of the pursuit movement and the saccade, leading to saccades going with the pursuit being faster, and saccades in the opposite direction of the pursuit being slower. Guan et al. (2005) found that the direction of the smooth pursuit target was also relevant, with higher peak velocities for saccades to horizontal targets heading towards the eye position, and lower than static target peak velocities for targets moving away from the eye position. These signals may have cancelled each other out, and the latter may be specific to saccades that start at head-centered coordinates, but the effect of a dynamic or static presentation of a complex visual stimulus did not adjust the main sequence in and of itself. 

Another factor that might affect these results is whether some of the saccades that make up the dynamic phase population are part of an optokinetic nystagmus behavior (OKN). Ter Braak showed in 1936 that there are two types of OKN with different slow phase and fast phase characteristics: look-OKN, where the subject is trying to get information from the visual stimulus, and stare-OKN, where the subject is just staring at a field of moving stimuli (Ter Braak, 1936). Stare-OKN has a lower gain during the slow phase and slower velocities during the quick phase (Kaminiarz, Konigs, & Bremmer, 2009). Most comparisons done by researchers have not controlled for this difference (Garbutt, Harwood, & Harris, 2001; Lappe & Hoffmann, 2000), and so it is not surprising that they found OKN to have lower peak velocities and longer durations. When the main sequences of saccades and the fast phase of look-OKN were compared, there was not a significant difference (Kaminiarz et al., 2009). Whereas this research was done on humans instead of macaques, because the visual information that is available during the OKN is important for navigation, we think that it is likely that the subjects are executing look-OKN rather than stare-OKN. 

Concerning our results, if the OKN fast phases are slower than normal saccades, and saccades during dynamic phases were also slower than during static phases, this would lead to an enhanced difference in the main sequences, which is not what we found. If on the other hand, slower OKN movements were mixed in with faster saccades during the dynamic phases, this could possibly lead to our results of finding no difference. This is unlikely for several reasons, the first being that OKN fast-phases are not usually very large, and so would not affect the bins at high amplitudes. It is also unlikely because the visual stimuli are rich with information, which suggests that it would generate look-OKN. This means that any OKN fast phases in primates during VE navigation should not be slower than saccades in the static phase. 

We observed that the monkeys spent more time looking at the screen during the Goals period, which suggests that they might have been more engaged during this period. There was a significant effect for the ANOVA, and the categorical difference was 60.4°/s, or 5% faster saccades during the Goals period. The difference between the two periods, along with the increased proportion of time with their eyes on screen, suggests that the monkeys were modifying their behavior in response to task demands. The most obvious difference between the two periods is the fact that the Navigation period is basically one of waiting before the subject gets to the decision point, and the Goals period has targets on the screen that are associated with rewards. 

When we only used Goals period saccades to compare against the Foraging saccades, there was no significant difference between main sequences. This suggests that when there is a directly rewarded object available in the environment (exploitation), then the saccades are faster compared to when the animal simply foveates objects out of curiosity or without a defined link to a reward (exploration). Galley wrote one of the first papers that attributed changes in the main sequence to “activation” (Galley, 1989; Galley 1998). He found that saccade velocities, so often measured at their ceiling, could then be found to drop as participants became less “activated”: drowsy, disinterested, or disengaged. More recently, Di Stasi has done significant research on this topic confirming the effects of drowsiness (Di Stasi et al., 2012, 2016), as well as finding effects of mental workload, where more complex tasks result in slower saccade velocities (Di Stasi et al., 2010; Di Stasi, Antolí, & Cañas, 2013). “Activation” may be the cause of the difference between the Navigation period (exploration behavior) and Goals period (exploitation behavior). In the Goals period and the Foraging task there is exploitation behavior, and it does not appear that any workload difference between the Goals period and the Foraging task is affecting the main sequence. 

Having found a possible explanation for the differences between the main sequences of all saccades in the VE tasks, we turned to also analyzing the classic Cued Saccade task. The first analysis replicated results from other studies that compared saccades to targets with saccades to a blank screen (Edelman & Goldberg, 2001). Edelman and Goldberg found that the size and presence of a target can affect the velocity of a saccade (Edelman & Goldberg, 2003). They found that a saccade to a point is 7% faster than to a vertex of a square and 17% faster than to the center of a square. Our results are similar, in that saccades to dots (VGS) were 18% faster than to nothing (NVGS). As there were far more saccades to blank space than there were to targets, our Cued Saccade task main sequence is largely comprised of these slow saccades. To do a comparison of matching target saccades against both the VE tasks resulted in a very low number of matches, so we compared them separately. Cued VGS were 7% faster than Goals period saccades and 8% faster than Foraging task saccades. This is the opposite finding from the initial analysis in Figure 4, and shows that saccades to small discreet targets were faster than saccades to targets in the VE tasks when properly matched. We also compared the slower saccades in VE, those during the Navigation period, to the nonvisually guided saccades (NVGS) from the Cued Saccade task. NVGSs were slower than saccades during the Navigation period. This could be because of the presence of visual stimuli and local landmarks at the end-point of the saccade during the Learning task. It is well documented that saccades to a region in space without a target are slower than to a target (Edelman & Goldberg, 2003; Edelman, Valenzuela, & Barton, 2006; Van Gelder, Lebedev, & Tsui, 1997). Alternatively, it could be because when the monkeys are engaged in the VE task, their level of arousal is higher, leading to faster eye movements; or it could be a combination of the two. 

Conclusion

In conclusion, our results demonstrate that when using dynamic display in VE and giving macaques the possibility to navigate and perform different tasks, the pattern of eye movements changes. The use of smooth pursuits to gather information is magnified relative to the use of only fixations. Moreover, the kinematics of eye movements also changes within the same VE to reflect differences between periods of exploration of the environment and exploitation of the rewards associated to certain targets. Importantly, we have proposed a method to classify eye movements in VE tasks that can be used and verified by future study in this exciting field. 

Acknowledgments

Funding sources: NSERC, CIHR, Reserach Chair in Autism (Ontario). 

Commercial relationships: none. 

Corresponding authors: Julio C. Martinez-Trujillo; Benjamin W. Corrigan. 

Email: ; . 

Address: Robarts Research Institute, University of Western Ontario, London, ON, Canada.