Seeing on the fly: Physiological and behavioral evidence show that space-to-space representation and processing enable fast and efficient performance by the visual system

Abstract

When we view the world, our eyes saccade quickly between points of interest. Even when fixating a target our eyes are not completely at rest but execute small fixational eye movements (FEMs). That vision is not blurred despite this ever-present jitter has seemingly motivated an increasingly popular theory denying the reliance of the visual system on pure spatial processing in favor of a space-to-time mechanism generated by the eye drifting across the image. Accordingly, FEMs are not detrimental but rather essential to good visibility. However, the space-to-time theory is incompatible with physiological data showing that all information is conveyed by the short neural volleys generated when the eyes land on a target, and with our faithful perception of briefly displayed objects, during which time FEMs have no effect. Another difficulty in rejecting the idea of image representation by the locations and nature of responding cells in favor of a timecode, is that somewhere, somehow, this code must be decoded into a parallel spatial one when reaching perception. Thus, in addition to the implausibility of generating meaningful responses during retinal drift, the space-to-time hypothesis calls for replacing efficient point-to-point parallel transmission with a cumbersome, delayed, space-to-time-to-space process. A novel physiological framework is presented here wherein the ability of the visual system to quickly process information is mediated by the short, powerful neural volleys generated by the landing saccades. These volleys are necessary and sufficient for normal perception without FEMs contribution. This mechanism enables our excellent perception of brief stimuli and explains that visibility is not blurred by FEMs because they do not generate useful information.

Introduction

We scan the world via a series of fast eye movements (“saccades”) that land our eyes on objects of interest ∼4 times a second. Interestingly, even between saccades (“fixational pauses”) our eyes are not completely at rest but execute miniature movements (fixational eye movements [FEMs]) consisting of occasional brief microsaccades, a continuous very-low-amplitude tremor, and a dominant, slow, low amplitude drift. In a normal scan (Figure 1), fixational pauses last ∼250 msec (Gersch, Kowler, Schnitzer, & Dosher, 2008; Boi, Polleti, Victor, & Rucci, 2017), during which time the visual system transmits, processes, and analyzes a great deal of information; we perceive the whole spatial tapestry in front of us, as well as its many elements. It has been an accepted truism that the extraordinary ability of the visual system to handle vast amounts of information during these brief pauses stems from its built-in parallel processing mechanisms. The two-dimensional (2D) retinal image is transmitted and processed in parallel, with minimal delays, through the complex retinal circuitry, the multi-layered lateral geniculate nucleus (LGN), and the early stages of the primary visual cortex (V1). Even the dominant physiological view that attributes our object perception to expert cells such as “face cells” and even “Gestalt cells” (see review by Spillman et al., 2023) relies on convergence and integration mechanisms emerging from the parallel organization of V1 cells. Despite the intuitive appeal of a 2D image leading through parallel pathways to our perception that retains the parallel nature of the outside world, there is an emerging consensus that during fixational pauses the drifting eye converts spatial information, at each retinal or V1 locus, into a space-to-time code. This transformation, it is argued, is beneficial and possibly even essential to good visibility (cf., Rucci, Ahissar, & Burr, 2018). A partial list of putative benefits includes reformatting a static spatial pattern into a spatiotemporal code (Ahissar, & Arieli, 2012; Rucci et al., 2018, Rucci, Iovin, Poletti, & Santini, 2007), enhanced high spatial frequencies (SFs), orientation and contrast discrimination (Rucci et al., 2018; Boi et al., 2017; Ahissar, & Arieli, 2012; Rucci et al., 2007), enhancement of feature extraction and estimation (Kuang, Poletti, Victor, & Rucci, 2012; Greschner, Bongard, Rujan, & Ammermuller, 2002), improving acuity and hyperacuity (Ratnam, Domdei, Harmening, & Roorda, 2017; Anderson, Ratnam, Roorda, & Olshausen, 2020; Intoy & Rucci, 2020), overcoming retinal inhomogeneity (Anderson et al., 2020), organizing retinal images (Lapin & Bell, 2023), and providing efficient coding for neuromorphic vision (Testa, Sabatini, & Canes, 2023). Others (Hohl & Lisberger, 2011; Pitkow, Sompolinsky, & Meister, 2007), while acknowledging that FEMs generate significant neural responses, suggest mechanisms that allow the visual system to overcome these responses. Some studies suggested that drift-generated responses may be useful in preventing image fading (Martinez-Conde, Otero-Millan, & Macknik, 2013; Ahissar, & Arieli, 2012; Engbert, Mergenthaler, Sinn, & Pikovsky, 2011). 

Figure 1.

 

A schematic depiction of eye movements during normal scanning. (A) Our eyes move between points of interest by fast, ballistic type movements—saccades. At each selected location the eyes pause for ∼250 msec. During each pause the eyes drift in a seemingly random fashion. (B) Horizontal eye movements during a fixational pause are illustrated. The pause is dominated by eye drift, and it ends when the eye saccades to a new target.

In this Perspective I argue that the functionality of the visual system is based, by and large, on its fast parallel processing abilities. Physiological and behavioral evidence show that incoming data are handled in such short times that introducing any serial processing stage would render information transmission too slow and ineffective to represent the normal modus operandi of the visual system during data acquisition. 

Biological evidence shows that the drifting eye cannot generate meaningful responses

Response to the landing saccade dominates the entire drift period

In monkey V1 cells, responses to landing saccades show a strong transient phase up to 80 to 90 msec into the fixational pause before settling slowly to a fairly high level throughout the pause (Figure 2; Gawne, & Martin, 2002; Kagan, Gor & Snodderly, 2008; Ruiz, & Paradiso, 2012). Also, visual evoked potentials ([VEPs] recorded at the occipital OZ electrode) initiated by landing saccades are consistent with sigle cell data in showing a strong response peaking at ∼80 msec and decaying to an above baseline level after ∼130 msec (Kaunitz et al., 2014). Using mice and pigs isolated retinae, Idrees, Baumann, Franke, Münch, & Hafed (2020), showed that responses to saccade-like displacements produced strong transient firing rates, and that these responses reduced sensitivity to new stimuli, well into the post “saccadic” interval. Thus, during fixational pauses, any slowly-accumulated weak responses that may be due to the drifting eye are negligable relative to the strong persistent volley generated by the landing saccade. 

Figure 2.

 

A schematic illustration of V1 cells responses to flashes and saccade landings (adapted from Gawne, & Martin, 2002; Kagan et al., 2008; Ruiz, & Paradiso, 2012). Responses to both stimulus types peak around 50 msec from saccade landing or flash onset and remain high throughout the fixational pause or the comparable flash duration. Similarities between responses to flashes and saccades are strong.

There is no time for drift-generated spikes

Given that the strong transient neural volley resulting from the landing saccade lasts at least 80 msec into the pause before starting to moderate (see Figure 2), and that preparation for the next saccade starts ∼100 msec before the end of the fixational pause (Rolfs, & Carrasco, 2012), there are only ∼70 msec, in a 250 msec pause, where drift may be effective. Now, it takes a 2’ drift to enable a 1′ receptive field (RF) to fully cross a 1′ spatial element. At 10′/sec drift velocity (cf., Figure 2, Ratnam et al., 2017), a 2′ drift lasts 200 msec, which is much longer than the 70 msec “effective” drift window. Other spatial elements/RFs interactions would require much longer intervals. So, even if somehow the meager drift-generated spikes could be discerned among the many saccade-generated ones, there is simply no time for the drifting eye to produce any meaningful response for even the smallest spatial elements. 

The erratic nature of the drift trajectory makes any space-to-time code impossible

Drift trajectory across a visual scene is not linear and predictable but rather resembling a Brownian motion (Engbert, & Kliegl, 2004; Cottaris, Wandell, Rieke, & Brainard, 2020), such that direction reversals and loops are often observed (cf. Figure 1 in Engbert, & Kliegel, 2004). Furthermore, even in a single subject repeatedly fixating the same target, saccade landing locations and drift trajectories differ between trials. Clearly no consistent space-to-time coding and decoding can be had under such conditions. 

Single cells response latencies and magnitude are quite variable

For the drifting eye to generate a reliable, consistent time code, the visual system must rely on the orderly, sequential, repeatable responses of the visual cells that slide across the various spatial elements. However, this is not the case. In V1 simple cells, for example, response latencies are quite variable for any given cell (Schmolesky et al., 1998, 66 ± 10.7 msec [µ ± std]; Bair et al., 2003, 52 ± 16 msec), as well as between same category cells (Bair et al., 2003). Moreover, Gur and Snodderly (2006) showed that response variability was particularly high for low response rates, which is the case for the very few spikes that may be related to the drifting eye. 

We must then conclude that the neural volley generated by the landing saccade carries all incoming information while drift mediated retina-image interactions of any kind cannot be useful. This conclusion is consistent with the demonstration of Resulaj, Ruediger, Olsen, & Scanziani (2018) that for some perceptual decisions, the first evoked spikes in V1 are sufficient. 

Curiously, for quite a long time all experimental and theoretical work on the presumed benefits of FEMs concentrated on drift while completely ignoring the much stronger neural effects of the landing saccade. Two fairly recent studies (Boi et al., 2017; Mostofi et al., 2020), though, did consider the perceptual affects of the pre-fixation saccadic high velocity sweep and suggested that, say, 3° to 5° saccades shape the image such that, at fixation start, very low SFs (<0.1 cyc/°) are enhanced. Consequently, fixational pauses can be divided into two (unspecified) intervals; in the first, low SFs are enhaned, whereas during the later, drift-dominated interval, high SFs are processed, resulting in “coarse-to-fine processing dynamics.” However, such an approach is untenable. It is well established, contrary to the authors’ scheme, that the strongest responses will be produced when a saccade lands the high acuity fovea on a high SF target, whereas landing on very low SF images will be very ineffective. For example, in the foveal representation, V1 cells with their very small, finely tuned receptive fields, respond vigorously to very high SF stimuli and are completely inhibited by low SF ones (cf., Gur & Snodderly, 1987; Gur & Snodderly, 2008). Furthermore, the authors’ analysis is based on the presumed continuity beween high velocity saccades and the very low velocity drift; a continuity that ignores the intervening flash-like 1- to 2-msec deccelaration that is saccade landing. 

No convincing evidence in support of the space-to-time theory

Practically all empirical evidence supportting the space-to-time theory comes from showing that performance under normal fixation is better than that under stabilized view where drift is eliminated. Unfortunaly, all studies, used either CRT monitors where each pixel is flashed with a sub-msec persistence time (Elze, 2010) once every 5–13 msec (Rucci & Desbordes, 2003; Rucci et al., 2007; Boi et al., 2017; Intoy & Rucci, 2020; Mostofi et al., 2020), or illuminated for sub-µs dwell times every 33 msec, using a scanning laser display (Ratnam et al., 2017). When using such displays, images are never really drifting across the retina but rather are flashed many times on a “frozen” retina. Consequently, the experimental comparisons are, in fact, not between normal vs. stabilized viewing but between many sub-msec flashes stimulating the retina at slowly changing locations (“normal view”) versus sub-msec flashes stimulating the retina repeatedly at one location (“stabilized”). Viewed in this context, there is no convincing evidence that long-duration fixational drifts enhance visual performance by contributing to high SF vision. Moreover, retinal stabilization of the pulsed display can potentially hamper visual performance through trivial mechanisms such as pigment bleaching, synaptic depression, or stabilization errors, that will not be present when an image shifts across the retina. 

In addition, in all cases, stimuli were presented for durations longer than those characterizing the saccade/drift cycle (cf., Rucci et al., 2007, 1000 msec; Segal et al., 2015, 2000 msec; Boi et al., 2017, 800 and 2300 msec), and in some studies, no saccades preceded drift or jitter (Greschner et al., 2002; Segal et al., 2015). 

In support of the above analysis, it is useful to look at a study (Keesey, 1960), where true stabilization was achieved by using a mirror attached to a contact lens. For three high SF targets (Vernier, fine line, gratings) and exposure durations ranging from 20 to 1280 msec, there was no differences in performance between stabilized and nonstabilized conditions. 

Seeing in a flash

Perceptual findings indicate that visual information transfer can be an ultra-fast process. There is ample evidence showing that stimuli presented for brief durations are accurately perceived, despite the absence of significant retinal motion on these timescales. Using stimuli lasting <50 µsec Greene and coworkers demonstrated that it was possible to recognize contours (Greene, 2007), letters (Greene & Visani, 2015), and even words and sentences (Greene, 2016). Stimuli <30 msec allowed categorization of various natural objects (Thorpe, Fize, & Marlot, 1996; Fabre-Thorpe, Delorme, Marlot, & Thorpe, 2001; Fize, Fabre-Thorpe, Richard, Doyon, & Thorpe, 2005; Mack & Palmeri, 2015), and color discrimination in monkeys (Stanford, Shankar, Massoglia, Costello, & Salinas, 2010). A mask presented 40 to 60 msec after stimulus onset did not prevent image categorization (Bacon-Macé, Macé, Faber-Thorpe, & Thorpe, 2005). Using <17 msec displays, subjects were able to detect, identify, and memorize complex objects such as faces (Rolls, Tovee, Purcell, Stewart, & Azzopardi, 1994; Keysers, Xiao, Földiák, & Perrett, 2001), and even a very short 120 fps presentation enabled subjects to detect repeating images (Thunell, & Thorpe, 2019). Zepp, Dubé, & Melcher (2021) used 8 msec displays enabling determination of Landolt C (∼6′ gap) orientation. Sperling (1960) showed that a 15 msec presentation enabled subjects to identify letters, and Zlatkova, Vassilev, & Mitov (2000) showed that subjects could detect small differences (8°–10°) in line orientations displayed for 10 msec. In the latter two studies, increasing stimuli duration to 500 msec did not improve perfomance. Even the minimalistic prediction of the space-to-time theory that, if nothing else, high SF visibility is necessarily achieved through FEMs is not supported by empirical findings. Packer and Williams (1992) showed that acuity for 50 and 60 c/° was within normal range and was not improved by increasing display duration during fixation, from 1 to 500 msec. Contrast sensitivity was not improved by extending display time of 100 c/° stimuli from 1 up to 2000 msec. In a similar vein, results by Gur and colleagues have shown that subjects can perceive high SFs with very short stimuli, including discrimination of <15’ wide faces displayed for a mere 33 msec (Gur, 2018), recognition of briefly flashed small doodles, contours, and faces (Gur, 2021), and hyper-acuity-level perception of 2 msec flashed stimuli (Hadani, Meiri, & Gur, 1984). The ability to correctly perceive brief stimuli is compatible with our capacity to quickly scan a complex scene by many short saccades (∼130 msec/saccade; Martin, Davis, Riesenhuber, & Thorpe, 2018). 

Flashes mimic landing saccades in onset dynamics and neuronal responses

Is the excellent visibility of brief stimuli a manifestation of normal acquisition and processing of visual information or is it a unique, esoteric phenomenon resulting from, say, the time-transients generated by pulsed displays? To answer this question, it is useful to compare the dynamics of saccade landing and flashes. To switch from a high angular motion velocity to an abrupt stop, takes the eye 1-2 msec (See Gawne, & Martin, 2002, Figure 1; Morales, Costa, & Woods, 2021, Figure 2). In experimental settings, although some flashes are generated by devices such as lasers (Ratnam et al., 2017) or LEDs (Greene,& Visani, 2015) with rise times in the µsec range, most flashed stimuli are displayed on computer monitors where rise times vary with refresh rate and stimulus size, and are usually >1 msec (Keyser et al., 2001 ∼3 msec; Gawne, & Martin, 2002 >2 msec; Kagan, Gur, & Snodderly, 2008, ∼1 msec; Crouzet, Kirchner, & Thorpe, 2010 ∼3 msec; Ruiz, & Paradiso, 2012, ∼1 msec; and Gur, 2018, 2021, >1 msec). The similarity between the dynamics of flashes and saccade landings indicates that our excellent perception of flashed information is not due to artificially generated time-transients but a direct result of the way our retinas are stimulated: in both cases there is a fairly abrupt interaction between the image and retinal cells, either caused by the sudden landing of the eye on a new object (natural viewing) or by the fast rise time of the screen-generated image. Thus, when a saccade lands the retina on an object, the result is very much flash-like. 

Physiological mechanisms underlying the excellent perception of briefly presented stimuli

Given the strong resemblance in stimulation dynamics between monitor-generated flashes and landing saccades, it is not surprising to find considerable similarities in physiological responses to flashes and saccades. In monkey V1 cells, responses to both stimuli peak at 40 to 60 msec and return to an above baseline level only ∼150 msec or more later (Figure 2; Gawne, & Martin, 2002; Kagan, Gur, & Snodderly, 2008; Ruiz, & Paradiso, 2012). Similarly, using voltage sensitive dyes, Seidemann, Chen, Geisler (2009), showed that V1 cells responded to brief stimuli, after a ∼25 ms latency, with a robust response, peaking at ∼100 ms. A recent publication (Kadosh, & Bonneh, 2022) showed that human visual evoked potentials generated by flashes are very similar in characteristic landmarks (amplitudes and polarities) to those evoked by landing saccades and microsaccades. The same work showed that the phenomenon of a saccade or a micorosaccade inhibiting the generation of the next microsaccade (oculomotor inhibition) is similar to the oculomotor inhibition caused by a flash. Idrees et al. (2020) demonstrated strong similarities in neural responses in mice and pigs retinal ganglion cells to saccade-like displacements and flashes. 

We can thus conclude that flashes and saccades impact the retina much the same way so that behavioral and physiological responses to flashed stimuli provide excellent evidence of the normal abilities of the visual system in the absence of FEMs. This means that short-duration stimuli that have been extensivly used in behavioral and physiological reseach are not merely a useful laboratory tool but provide direct insights into normal visual mechanisms. 

Demos

The attached demos (see Supplementary Materials) allow the reader to experience firsthand the ease with which various flashed stimuli are perceived. A wide range of images and SFs are available. 

Why is vision not blurred by FEMs during fixational pauses?

To fixate a target, the eyeball must be immobilized for ∼250 msec by the simultaneous contraction of the three pairs of ocular muscles. Since this is a biological system, some degree of imperfection, or noise, is expected, and is manifested by the eyes drifting for 2-3′ during a fixational pause. This ability of maintaining fixation is quite remarkable, with no need to suggest that fixational drift results from a functional necessity. However, the discovery that even during fixational pauses our eyes are not at rest raised the question of why vision is not blurred during these pauses. This perennial question motivated, in part, the space-to-time approach which explained that our perception is achieved through FEMs, so not only they do not hinder vision, but they are the mechanism that enables it. Our analyses suggest a simpler, straightforward explanation: given that all relevant visual information is carried by the short neural volleys produced by the landing saccade, the neural responses that may be generated by the succeeding retinal jitter are not visually informative and thus do not blur the image. 

Fast transmission and processing are enabled by keeping a strictly spatial, one-to-one representation

Viewing and comprehending the world entails transmission and processing of enormous amount of information in a very short time. We may view a scene for a few hundred msec and then shift our eyes briefly to view another one. Each scene may include many objects, each containing many elements, and yet we perceive and characterize each object, its relationship to other objects, its location in space and more. This is made possible by preserving the 2D parallel spatial layout of the retinal image all through the complex retinal circuitry, LGN, and the initial stages of V1. From the 2D cortical representation of the retinal image, a three-dimensional representation of the world is built as evidenced in our three-dimensional parallel perception. The proponents of the space-to-time view suggest that the one-to-one representation of space found in V1 is temporarily replaced by a one-to-few representation timecode generated by the slow, protracted process of the retina drifting across the image during fixation. Somewhere, somehow, this timecode must be decoded into a one-to-one spatial representation when reaching perception. Thus, in addition to the implausibility of generating meaningful neural responses during retinal drift, the space-to-time hypothesis calls for abandoning the efficient, almost instantaneous parallel transmission in favor of a cumbersome space-to-time-to-space process. The evidence presented in this Perspective shows that adding a serial processing stage is untenable and that the parallel spatial input is kept as such and is delivered to the perceiving stage in a flash-like manner (cf. Rousselet, Thorpe, & Fabre-Thorpe, 2004) by the volleys generated by the landing saccades. Indeed, the ability to quickly scan a complex scene by many saccades with short acquisition and processing times is of considerable evolutionary and social-behavior value. 

Acknowledgments

Commercial relationships: none. 

Corresponding author: Moshe Gur. 

Email: [e-mail]. 

Address: Department of Biomedical Engineering, Technion—Israel Institute of Technology, Haifa 32000, Israel.