Motion-generated optical information allows event perception despite blurry vision in AMD and amblyopic patients

Abstract

Events consist of objects in motion. When objects move, their opaque surfaces reflect light and produce both static image structure and dynamic optic flow. The static and dynamic optical information co-specify events. Patients with age-related macular degeneration (AMD) and amblyopia cannot identify static objects because of weakened image structure. However, optic flow is detectable despite blurry vision because visual motion measurement uses low spatial frequencies. When motion ceases, image structure persists and might preserve properties specified by optic flow. We tested whether optic flow and image structure interact to allow event perception with poor static vision. AMD (Experiment 1), amblyopic (Experiments 2 and 3), and normally sighted observers identified common events from either blurry (Experiments 1 and 2) or clear images (Experiment 3), when either single image frames were presented, a sequence of frames was presented with motion masks, or a sequence of frames was presented with detectable motion. Results showed that with static images, but no motion, events were not perceived well by participants other than controls in Experiment 3. However, with detectable motion, events were perceived. Immediately following this and again after five days, participants were able to identify events from the original static images. So, when image structure information is weak, optic flow compensates for it and enables event perception. Furthermore, weakened static image structure information nevertheless preserves information that was once available in optic flow. The combination is powerful and allows events to be perceived accurately and stably despite blurry vision.

Introduction

Many eye diseases cause uncorrectable reduction in visual acuity (VA) and/or contrast sensitivity (CS; Jobling, Mansfield, Legge, & Menge, 1997; Leat, Legge, & Bullimore, 1999). At the clinic, a patient who reports experiencing blurry vision first has their VA and CS tested using still images (e.g., with Snellen or Pelli-Robson charts; West et al., 2002). In addition, to assess the effect of treatment and rehabilitation, improvement in static vision is measured using, for example, static VA and CS tests (Hooper, Jutai, Strong, & Russel-Minda, 2008). The emphasis on static image–based vision is because of its assumed relevance to the detection of edges and boundaries in images (Marr, 1982/2010), which is presumed to underlie important visual functions such as object perception, scene recognition, and action guidance. For instance, image-based vision is tested for obtaining a driver's license using static images. 

Although static image–based vision is relevant, it is not the whole story. In natural viewing environments, opaque surfaces reflect light and project image structure information to an observer. This image-based information allows perceiving objects in a stationary environment. When observers and/or objects move, the resulting relative motions yield continuous transformations in image structure called “optic flow” (Gibson, 1979/1986). Optic flow is generated by motion and provides information about both the three-dimensional (3D) layout of surrounding surfaces and the relative motion between those surfaces and the observer (Koenderink & van Doorn, 1987; Lind, 1996). The speed of optic flow covaries with the distance between surfaces and the point of observation and thus provides a relative depth map of the surroundings (Nakayama & Loomis, 1974). It also varies with the relative speeds of motion either of the observer or of objects in events (Lind, 1996). Therefore, optic flow also enables event perception, which includes identifying objects in events (Bingham, 1995; Bingham & Wickelgren, 2008). Johansson (1973) demonstrated this using displays filmed in the dark in which only small lights attached to moving objects (e.g., on joints of a walking person) were visible. Image structure was made uninformative by the point light technique, which reduced the images to extremely low spatial frequencies. Given any static frame from the film, naïve observers saw only bright dots on a dark background and were unable to identify anything meaningful. However, when the film was played so that motion information was available, the event was perceived immediately. Since then, this has been verified in perception of a variety of inanimate and animate events (Bingham, Rosenblum, & Schmidt, 1995; Jansson, Bergstrom, & Epstein, 1994). Therefore, when perceiving events, objects may be specified by image structure and/or optic flow information. When image structure becomes hard to detect, either as a result of weakened signals (e.g., low spatial frequencies in visual stimuli) or insensitive detectors (e.g., blurry vision caused by eye problems), events and objects can still potentially be perceived using optic flow. 

Both optic flow and image structure provide information about events and the constituent objects. Naturally, the visual system uses both. On one hand, optic flow information is strong in specifying 3D spatial relations and enables event perception, but it is transient. It varies in quality with the relative speeds of motion and becomes unavailable when motion stops (or, more generally, falls below threshold). On the other hand, image structure is weaker in specifying 3D spatial relations, especially when observers have blurry vision and/or when the quality of visual stimuli is poor. For example, the point lights in Johansson's (1973) display would not be perceptible as a walking event in a single static frame extracted from the film. However, image structure is stable. It remains available to an observer so long as objects are visible. 

In an environment where objects and/or observers move, both optic flow and image structure information become available. They are intrinsically related and largely symmetric (i.e., specifying the same events): Optic flow carries one structured image into the next structured image, or one image “flows” to the next. The relation between optic flow and image structure, in part, could be cast as a calibration of weak image structure information about 3D by the more powerful optic flow information. Just as knowledge of the result of an action provides feedback for visually guided actions and calibrates the otherwise ambiguous perceptual information used to guide those actions to be accurate (e.g., Bingham & Mon-Williams, 2013), optic flow specifies spatial layout and can be used to calibrate the otherwise ambiguous image structure to be accurate in specifying events. After motion stops, only image structure information is available. Unlike in the premotion phase, now (having been calibrated) the poor image structure contains information for spatial relations, affords event perception, and continues to allow event perception so long as the blurry images remain available. So, we say image structure is the information, external to an observer, that once calibrated by optic flow, preserves the spatial relations (and hence the visual events) that were specified by optic flow. Observers need not store previously perceived events with ongoing optic flow in the head. Instead, they can extract events from the image structure information in real time. In this sense, calibrated image structure forms an embodied memory system that facilitates recall with long time delays (see Pan, Bingham, & Bingham, 2013, for details). 

The effectiveness of interacting optic flow and image structure information is especially relevant to observers with blurry vision, where the access to high spatial frequencies in images is limited but the detection of optic flow, which takes low spatial frequencies, is largely intact (Smith & Snowden, 1994; Straube, Paulus, & Brandt, 1990). Motion processing has been shown to compensate for the loss of high spatial frequencies in images. For example, in visual signals with low temporal and spatial frequencies, the detection of contrast is bad; when temporal frequency is increased, contrast detection improves, for high and low spatial frequency visual signals (Robson, 1966). Also, it has been shown that when an image moves, motion had a deblurring effect, making observers' blur discrimination thresholds increase (Burr & Morgan, 1997), although in fact when images began to move, high spatial frequencies were attenuated and blur increased (Pääkkönen & Morgan, 1994). On the other hand, apparent motion was more robust when spatial frequencies were low. Observers in Chang and Julesz's (1983) study tolerated larger displacement between images and identified properties of apparent motion from random-dot images with lower than higher band-pass filters. 

Thus, would the combination of poor image structure information and intact optic flow processing help observers with blurry vision perceive events in an accurate and stable fashion? Based on previous literature that showed that the visual system is sensitive to moving images with blur, we hypothesized that observers with blurry vision should not be able to identify events from static images (whether the images themselves were blurred or not) but that they should be able to identify visual events from sequences of images containing motion. Importantly, after identifying events with ongoing motion, observers should be able to recognize the events from static image structure, both immediately after motion stops and later with time delay, implying that calibrated image structure preserves events specified by optic flow. In three experiments, we tested this hypothesis with two types of patients who experience blurry vision, namely, patients with age-related macular degeneration (AMD) and younger patients with amblyopia. The AMD patients had moderate low vision and viewed the experimental display with both eyes. The amblyopic patients had severely undermined VA and CS in the amblyopic eyes, and they viewed and identified the experimental display with their amblyopic eyes. In Experiments 1 and 2, blurry events were used; AMD, amblyopic, and control participants attempted to identify events from either static blurry images or moving sequences of blurry images. In these experiments, we expected similar performance for clinical and control participants. In Experiment 3, unblurred events were used, and we adjusted the display size so that amblyopic participants were unable to identify events from the clear static images. Then, we tested how they performed when seeing these images in motion. In this experiment, we expected different performance for clinical and control participants. In particular, controls viewed the same display sizes as amblyopic participants and were readily able to identify the clear static images. 

Experiment 1: Perceiving blurry events, AMD patients

Methods

Participants

Sixteen adults participated in the study; eight with dry or wet AMD and eight age-matched controls with normal vision. The normally sighted participants were recruited from the local community or through a relationship with another participant. The AMD participants were recruited from the Indiana University School of Optometry Primary Care and Vision Rehabilitation Service clinics. 

Informed consent was obtained from each participant after the nature and possible consequences of the study were described. The study followed the tenets of the Declaration of Helsinki and was approved by the Institutional Review Board of Indiana University. Participants were compensated at the rate of $10/hr. 

Materials

Each participant completed a set of vision tests, a cognitive screen and a perception experiment. The vision tests included a test for binocular VA, a test for binocular CS, and a test for binocular visual field size (VFS). The VA was assessed using an Early Treatment Diabetic Retinopathy Study (ETDRS) acuity chart (Ferris, Kassoff, Bresnick, & Bailey, 1982), which was trans-illuminated to approximately 100 cdm⁻² and scored in log MAR using the method of Bailey and Lovie (1976). The CS was assessed using the Evans Letter Contrast Test. The Evans Letter Contrast Test, a chart-based assessment, consists of two charts that are trans-illuminated to approximately 85 cdm⁻². The first and second chart contain 12 triplets and four triplets, respectively, of Sloan letters that are 20/630 in size. Like the Pelli-Robson letter CS Chart (Pelli, Robinson, & Wilkins, 1988), the Evans Letter Contrast Test has two sets of triplets per line, and the contrast decreases with each successive triplet. Each correctly identified letter is assigned a score of 0.05 log CS, with the exception of the first triplet, which was 0 log CS. The binocular threshold log CS was determined using the method of Elliott and colleagues (Elliott, Bullimore, & Bailey, 1991; Elliott, Whitaker, & Bonette, 1990). Finally, kinetic perimetry was performed on each subject binocularly using an Octopus perimeter (III4e target on a background luminance of 10 cdm⁻²) to assess the binocular VFS along 12 meridians from radii of 70° to 90° vertically and 90° horizontally. The average binocular VF extent (radius) was computed for each subject by averaging the VF extent along each meridian. The AMD subjects were instructed to use eccentric fixation, presumably with their preferred retinal locus (PRL), during all of the binocular vision assessments. 

A cognitive screen was conducted to ensure that participants had adequate cognitive function to complete the perception experiment. Using the Mini-Mental State Exam (MMSE; Folstein, Folstein, & Mchugh, 1975), we screened participants for their basic information-processing capabilities, short-term memory, executive function, attention, and mental flexibility. 

Visual stimuli in the perception experiment consisted of eight everyday events, recorded with a Canon digital camera. The events were a man shooting a basketball, a woman bowling, a woman washing dishes at a kitchen sink, a dog running in a backyard, a woman pouring tea from a thermal flask into a cup on a table, a man and woman dancing tango, two teams playing soccer on a field in a stadium, a man walking toward the camera and waving his hand. 

All videos were then processed with Adobe Premier Pro CS5. The final videos were silent and black and white with reduced contrast and high blur. Gaussian blur with a 15-pixel radius in the horizontal and vertical directions was applied to the videos. The cutoff frequency of the Gaussian blur filter was 0.02 cycles/pixel. We selected 20 still frames from each processed video clip as test stimuli in the experiment (see Figure 1 for example frames). The frames were assembled in a Java applet (display size: 21 cm × 14 cm, approximately 30° × 20° visual angle at a viewing distance of 40 cm). Given the resolution of the images (720 × 480), the display size (21 cm × 14 cm), and the 40-cm viewing distance, the frequency cutoff of 0.02 cycles per pixel converted to 0.48 cycles per degree visual angle in the final stimulus set. The Java applet allowed the frames to be displayed one at a time (in Conditions 1, 4, and 5), in sequence with white frames (serving as motion masks) in between (in Condition 2) and without white frames in between (in Condition 3). The stimuli were displayed on a 19-in. Dell monitor with a refresh rate of 60 Hz. There was a text box below the display (part of the Java applet) for typing descriptions of the displayed images. 

Figure 1

 

Examples of static blurry images of familiar events, which were the stimuli used in Experiments 1 and 2.

Procedures

This study required two sessions. The cognitive screen, vision tests, and the first four conditions of the perception experiment were performed in the first session. The fifth condition was completed in the second session five days later. 

In the first session, participants first completed the visual tests and the cognitive screen. A trained experimenter measured each participant's binocular VA, CS, and VFS in a private room. The experimenter also conducted the MMSE and the Trail Making Test–Part B to test the participant's cognitive function. Participants were required to score 24 or above on the MMSE to qualify for the perceptual experiment. All 16 participants met this requirement. 

Next, participants did the first four conditions of the perception experiment. Participants were seated in front of a computer screen at a viewing distance of 40 cm. Seat height was adjusted so that participants' eyes were aligned with the center of the screen. An experimenter explained that that the participant would see blurred images and animations depicting common everyday events. The task was to describe the event in each display. The experimenter typed out the participant's responses in the text box below the display. Participants were encouraged to use eccentric viewing, when needed. 

In the first condition, three of 20 static frames from each of the eight events were randomly selected to present to participants, one at a time (a total of 24 trials). In this condition, there was only static image structure. See Figure 1 for an example. In the second condition, the 20 frames from each video were played in the order they appeared in the natural events, with a white screen inserted after each frame, serving as a motion mask. The duration of each frame was 500 ms and that of the white screen (the motion mask) was 2000 ms. In this condition, there was complete image information, but no motion information. In the third condition, the 20 blurred frames taken from each video were played in sequence without the white screens (motion masks), providing both image structure and motion-based information. Fourth, after the blurred images had been viewed in the context of motion, participants were given 24 static blurred images (three randomly selected frames from eight processed videos) and again asked to identify the events. The stimuli and procedure were identical to those in the first condition. 

Finally, five days later, participants returned for a final test. First, they verbally reported the events they could remember having previously identified. Then, participants observed 24 blurred images from eight events and described the events, just as they had in Conditions 1 and 4. The experimenter recorded their responses. 

While participants were describing the stimuli, in Conditions 1, 4, and 5, the static images were continuously displayed on the monitor. In Conditions 2 and 3, ordered frames were played in loops. 

An experimental demo (with instructions) can be downloaded from http://www.indiana.edu/∼palab/Resources/Demos/SLAEblur4_RunExp.zip. For additional information, visit: http://www.indiana.edu/∼palab/research.php and click to expand “Perception and Embodied Memory”.) 

Data processing

There were 24 trials (three for each of eight events) in Conditions 1, 4, and 5 and eight trials (for the eight events) in Conditions 2 and 3. Thus, every participant completed 88 trials. One trial was excluded because of experimenter error, making a total of 1,407 trials for 16 participants. 

The data were descriptions of the events. The descriptions were coded as correct or incorrect by two raters, who had watched both the blurry displays and the original untreated videos. In all three experiments, the raters did not know whether a certain set of responses was from a normally sighted or visually impaired observer. A description was coded as correct if it captured the essence of an event. (For example, one of the events was a woman sitting at a table, pouring tea from a thermal flask into a cup and drinking from the cup. A correct response would have to cover the meaning of pouring liquid and drinking. Details like “a woman” [versus “a person”] or “a thermal flask” [versus “a container”] were not required.) Raters did not use lists of key words or other explicit rules for coding. The two raters coded independently, and their coding was the same in 1,364 of 1,407 trials, making the interrater reliability 97% (assessed using the “joint probability of agreement” method; Uebersax, 1987). In the 43 trials in which the raters coded differently, one rater's coding was randomly picked. 

Results

Eight AMD participants and eight normally sighted participants completed Experiment 1. There was no significant age difference between the two groups, t(14.0) = 0.097, p = 0.92. The vision tests revealed that the AMD participants had lower binocular VA, t(14) = 2.99, p = 0.01, and binocular CS, t(14) = −3.17, p = 0.007, than the normally sighted participants. No significant difference in the average binocular VF extent (radius) was found between the AMD and age-matched normally sighted participants, t(14) = 1.03, p = 0.32. None of the participants, including the AMD subjects, had any central binocular scotomas. See Table 1. 

Table 1

 

Summary of participants with AMD in Experiment 1. Notes: Results listed as mean ± 1 SD.

 

In the perception test, we investigated if motion could calibrate blurred images to allow subsequent accurate perception of events despite limited image information. Our results showed that this was indeed true for both normally sighted observers and the AMD observers. 

To show that motion-generated information aided event perception with limited image details, we performed an omnibus mixed-design analysis of variance (ANOVA), comparing proportions of trials correctly perceived across different conditions and events (within-subject factors) for the AMD and normally sighted participants (between-subject factor). First, there was no difference in performance between the two types of observers, F(1, 14) = 0.008, p = 0.93. See Figure 2. Performance differed across conditions, F(4, 56) = 89.7, p < 0.001, with a higher rate of recall in the postmotion conditions of 4 and 5 than in the premotion conditions of 1 and 2. The factor of “event” was also significant, F(7, 98) = 10.37, p < 0.001, which suggested that some events used in the experiment were harder to perceive than others. In fact, although all eight events were equally well recognized in the motion condition, there were four events that had higher initial recognition rates (in Condition 1) than the rest. This was reflected as a significant event-condition interaction, F(28, 392) = 3.28, p < 0.001. Nonetheless, for every event tested, recognition rates in postmotion conditions were higher than in the premotion conditions. Hence, we shall focus on comparing performance across conditions. 

Figure 2

 

Performance for the AMD and normally sighted participants in five conditions. Error bar = 1 SE. Visual stimuli used in each trial: Condition 1: single static frame of blurry image; Condition 2: 20 frames of blurry images with motion masks; Condition 3: 20 frames of blurry images with motion; Condtions 4 and 5: single static frame of blurry image.

In Condition 1, events were perceived correctly in only 65 of 384 trials, or 16.9% (95% confidence interval [CI] = [13%, 21%]). In Condition 2, where sequences of images were available, performance did not improve, and only 19 of 128 trials, or 14.8% (95% CI = [9%, 21%]), were identified. So, regardless of the number of images, static image structure did not yield successful event perception. 

When both static image information and motion information were available in Condition 3, events were easily perceived, and the rate of correct identification was 82% (or in 105 of 128 trials among all participants, 95% CI = [75%, 89%]). Comparing performance in Condition 2 and in Condition 3, everything else being equal, the added motion information yielded a significant and robust improvement in event perception, F(1, 15) = 249.86, p < 0.001. 

Once the images had been calibrated by motion, the events represented in the blurred images were readily perceptible. In postmotion Condition 4, with the same images as in Condition 1, the rate of identification was 58.4% (224 of 383 trials, 95% CI = [54%, 63%]), significantly better than in Condition 1, F(1, 15) = 84.4, p < 0.001. Given that the only difference between Conditions 1 and 4 was previous availability of motion-based information, the significant contrast in performance suggests that the effect of motion was preserved in the blurred image structure. 

Finally, the participants were tested again after five days. In free recall, participants described 45 of 128 events (16 participants and eight events for each). Then, the rate of correct identification in Condition 5 (which was same as Conditions 1 and 4) was 56% (214 of 384 trials, 95% CI = [51%, 61%]). Performance was significantly better than in Condition 1, F(1, 15) = 64.0, p < 0.001, and was not worse than in Condition 4, which occurred immediately after Condition 3 (motion calibration), F(1, 15) = 0.50, p = 0.50. Furthermore, the high rate of identification during retest was not merely a function of whether participants had remembered the events that they had perceived five days before: The rate of event identification during retest was 38% for events that participants had failed to recall. This rate was significantly higher than the rate of identification in Condition 1 (i.e., 16.9%), t(435) = 5.74, p < 0.001, when participants first saw the blurred images. Instead, the rate of identification in this retention test was more closely related to performance in Condition 3, where blurred images were presented in the context of motion. Specifically, if participants did not perceive the events correctly in Condition 3, their rate of event identification in the retest was 15.9%, which was as low as that in Condition 1. These results showed that motion-generated information was crucial for perceiving events and that events perceived with ongoing motion were preserved in the blurry image structures and not held only in memory (that is, memory-in-the-head, as traditionally construed). 

Experiment 2: Perceiving blurry events, amblyopic patients

Methods

Participants

Sixteen participants (age range 13–29 years) completed this study: eight with amblyopia (four females; seven were right eye amblyopic, one was left eye amblyopic; seven were anisometropic and one was both anisometropic and strabismic) and eight age-matched controls with normal vision. The normally sighted participants were recruited from the local community. The amblyopic participants were recruited from Sun Yat-sen University Zhongshan Ophthalmic Centre. (Six other patients started the experiments but never finished because of no show during retest. Their data were excluded.) 

Informed consent was obtained from adult participants or parents of minors after the nature and possible consequences of the study were described. The study followed the tenets of the Declaration of Helsinki and was approved by the Institutional Review Board of Sun Yat-sen University. Participants were compensated at the rate of ¥50/hr. 

Materials

Each participant completed a set of vision tests (VA, CS, and VF), a cognitive screen, and a perception experiment. All were performed monocularly on the amblyopic eye for the clinical group or on the dominant eye for the control group. The VA was assessed using a tumbling E ETDRS acuity chart with luminance of 320 cd/m² (Chen et al., 2016). The CS was assessed using two-dimensional Gabor grating patch of 2° on a 17-in. Dell CRT monitor (refresh rate = 85 Hz). Three-down-one-up staircase was applied to measure the contrast threshold at 79.3% (Chen et al., 2016). To be comparable with Experiment 1, the CS was assessed at a spatial frequency of 1 cycle/°. Humphrey 750II visual field perimetry (Carl Zeiss, III target on a white background, central 30°, SITA-standard strategy) was assessed in both normal controls and amblyopic observers.¹ Mean deviation was obtained after each assessment. As in Experiment 1, we screened participants for their cognitive capacity, using a Chinese translation of the Mini-Mental State Exam (MMSE, Chinese translated version, Chiu et al., 1998). 

Visual stimuli in the perception experiment consisted of eight everyday events, recorded with a Canon digital camera. The events were children skipping ropes, four people playing badminton, a woman bowling, a dog running in a backyard, a mother and a child playing a clapping game, a man and woman dancing tango, a hair dresser styling a lady's long hair, and a man walking toward the camera and waving his hand. Three clips were identical to those used in Experiment 1, and five were new. The filming and processing of the video clips were identical to that in Experiment 1. The final visual stimuli had resolution of 720 × 480 pixels, display size of 21 cm (W) × 14 cm (H), and Gaussian blur of 0.02 cycles/pixel cutoff frequency, and they were displayed on a 27-in. LED monitor with resolution of 1,920 × 1,080 pixels and refresh rate of 120 Hz. 

Procedures

The procedures of this experiment were similar to those in Experiment 1 except that in this experiment, amblyopic participants viewed with their blurry eye and normal controls viewed with their dominant eye. 

During the first experimental visit, a trained experimenter measured each participant's binocular VA, CS, and VFS in a private room. The experimenter also conducted the MMSE (Chinese version, Chiu et al., 1998) to test the participant's cognitive function. All 16 participants scored above 24 on the MMSE and qualified for the perceptual experiment. 

In the perception experiment, the experimental task, conditions, and orders were identical to those in Experiment 1. During the first visit, participants performed Conditions 1 to 4, and they revisited the laboratory after five days to complete a free recall and an identification condition (Condition 5). 

Data processing

There were 16 participants, and each performed 24 trials (three for each of eight events) in Conditions 1, 4, and 5 and eight trials (for the eight events) in Conditions 2 and 3. Thus, altogether, there were 1,408 trials. Two raters independently coded event descriptions (using the same coding standard as in Experiment 1), and their coding was the same in 1,291 of 1,408 trials, making the interrater reliability 91.7% (assessed using the “joint probability of agreement” method; Uebersax, 1987). In the 117 trials in which the raters coded differently, one rater's coding was randomly picked. 

Results

In Experiment 2, we studied eight amblyopic participants and eight age-matched normally sighted participants. By design, there was no significant age difference between the two groups, t(10.7) = −0.38, p = 0.71. The vision tests revealed that the amblyopic participants had lower binocular VA, t(7.22) = 3.30, p = 0.013, than the normally sighted participants. There was no significant difference in binocular CS, t(9.74) = −0.38, p = 0.72. The group average mean visual field defect (MD) of the amblyopic observers was worse than the controls, t(9.92) = −2.33, p = 0.04. (This was probably because one amblyopic observer had extremely low MD.) None of the participants, including the amblyopic subjects, had any central scotomas. See Table 2. 

Table 2

 

Summary of participants in Experiment 2. Notes: Results listed as mean ± 1 SD.

 

In the perception test, we investigated if motion could calibrate blurred images to allow subsequent accurate perception of events despite limited image information. Our results showed that this was indeed true for both normally sighted observers and the amblyopic observers. An omnibus ANOVA was performed, comparing proportions of trials correctly perceived across different conditions and events (within-subject factors) for the amblyopic and normally sighted participants (between-subject factor). First, there was no difference in performance between the two types of observers, F(1, 14) = 1.75, p = 0.21. See Figure 3. Performance differed across conditions, F(4, 56) = 71.8, p < 0.001, partial η² = 0.84, with a higher rate of recall in the postmotion Conditions 4 and 5 than in the premotion Conditions 1 and 2 for both amblyopic and normal participants. The factor of “event” was also significant, F(7, 98) = 6.7, p < 0.001, partial η² = 0.32, which suggested that some events used in the experiment were perceived in a different pattern than others. However, the interaction between event and condition was not significant, F(28, 392) = 1.32, p = 0.129. So events did not affect performance in the five conditions differentially. Hence, in the subsequent analysis, we collapsed data collected from amblyopic patients and normal controls and focused on the effect of condition. 

Figure 3

 

Performance for the amblyopic and normally sighted participants in five conditions of Experiment 2. Error bar = 1 SE. Visual stimuli used in each trial: Condition 1: single static frame of blurry image; Condition 2: 20 frames of blurry images with motion masks; Condition 3: 20 frames of blurry images with motion; Condtions 4 and 5: single static frame of blurry image.

In Condition 1, events were perceived correctly in only 38 of 384 trials, or 9.90% (95% CI = [6.91%, 12.9%]). In Condition 2, in which sequences of images were available, performance did improve somewhat, but only 32 of 128 trials, or 25% (95% CI = [17.5%, 32.5%]), were identified. In these first two conditions, identification performance significantly improved as more frames of blurry images were available. 

When both static image information and motion information were available in Condition 3, events were easily perceived and the rate of correct identification was 69.5% (or in 89 of 128 trials among all participants, 95% CI = [61.5%, 77.5%]). Everything else being equal, the added motion information in Condition 3 yielded a significant and robust improvement (by ≈ 45%) in event perception as compared with Condition 2, F(1, 15) = 95.37, p < 0.001. 

In Condition 4, the rate of identification was 63.5% (244 of 384 trials, 95% CI = [58.7%, 68.4%]). Comparing Condition 4 with Condition 1, the test stimuli were identical, but performance in the postmotion Condition 4 was significantly better (by ≈ 54%), F(1, 15) = 107.6, p < 0.001. Therefore, the effect of motion in specifying visual events was preserved in the blurred images, allowing events to continue to be perceived after motion stopped. 

Finally, the participants were tested again after five days. In free recall, participants described 58 of 128 events (or 45%). Then, the rate of correct identification in Condition 5 was 59.6% (229 of 384 trials, 95% CI = [54.7%, 64.5%]). Performance was significantly better than in Condition 1, F(1, 15) = 77.9, p < 0.001, and was not significantly worse than in Condition 4, which was tested during the first session immediately after the motion condition, F(1, 15) = 1.46, p = 0.25. Furthermore, the high rate of identification during retest was not merely a function of participants remembering the events that they had perceived five days before: The rate of event identification during retest was 36.7% among events that participants had failed to recall. This rate was significantly higher than the rate of identification in Condition 1 (i.e., 9.90%), t(299) = 7.30, p < 0.001, when participants first saw the blurred images. Instead, the rate of identification in this retention test was more closely related to performance in Condition 3, where blurred images were presented in the context of motion. Specifically, if participants had not perceived the events correctly in Condition 3, their rate of event identification in the retest (21.3%) was significantly lower than if they had identified the events in Condition 3 (76.4%), t(228) = −11.9, p < 0.001. 

Experiment 3: Perceiving clear images, amblyopic patients

In the first two experiments, all participants viewed images that were extremely blurry and in which events were difficult to identify without motion-based information being available. Thus, similar performance was expected for both clinical and control participants. In Experiment 3, however, participants viewed clear, unblurred images. We now expected different performance by amblyopic patients and controls. We tested the former participants in conditions were they were unable to identify events in static images. We tested the latter participants in the same conditions but expected that they would be well able to identify the events. The question was whether the subsequent addition of motion-based information would enable the amblyopic patients to identify the events. 

Methods

Participants

Twelve participants (age range 14–32 years) completed this study: eight with amblyopia (two females; four were right eye amblyopic, four were left eye amblyopic; six were anisometropic, one was strabismic, and one was both) and four age-matched controls (two females) with normal vision. The normally sighted participants were recruited from the local community. The amblyopic participants were recruited from Sun Yat-sen University Zhongshan Ophthalmic Centre. 

Informed consent was obtained from adult participants or parents of minors after the nature and possible consequences of the study were described. The study followed the tenets of the Declaration of Helsinki and was approved by the Institutional Review Board of Sun Yat-sen University. Participants were compensated at the rate of ¥50/hr. 

Materials

Each participant completed the same vision tests and cognitive screen as in Experiment 2. In the perception experiment, the visual stimuli consisted of the same events as in Experiment 2 (resolution of 720 × 480 pixels). However, the black-and-white images of events were not blurred. 

Procedures

As in Experiment 2, amblyopic participants viewed with their blurry eye and controls with normal vision viewed with their dominant eye. In addition, stimuli display size, in terms of degrees visual angle, varied between participants. We adjusted the viewing distance for each amblyopic observer so that each observer was unable to detect events from the unblurred static images. See below for details. 

During the first experimental visit, participants were tested for their VA, CS, and VF on the amblyopic (for the clinical group) or dominant (for the control group) eye. They also performed the MMSE (Chinese version, Chiu et al., 1998), and all scored 24 or above and qualified for the perceptual experiment. 

Next, amblyopic participants performed several pre-experimental trials to determine the effective display size (by varying the viewing distance and/or the onscreen size of the visual stimuli) at which they began not to be able to identify events in the clear static images. We used the method of adjustment to find each individual's threshold. Specifically, a set of 80 black-and-white images depicting daily events (resolution: 720 × 480 pixels; original onscreen display size: 21 cm × 14 cm) were displayed on a 27-in. LED monitor (resolution: 1,920 × 1,080; refresh rate: 120 Hz) one at a time. (We put the 80 images in a Java applet so that their presentation order was randomized and the image size could be easily adjusted by dragging an image with fixed width-height aspect ratio.) These events were different from the stimuli used in Experiments 1 or 2, but they also constituted familiar daily situations such as a dinner table with dishes and food, ships docked at a pier, a person sitting at her desk, and so forth (Figure 4). Amblyopic participants were seated and looked at an image with the blurry eye and reported what she or he saw. For each amblyopic participant, the experimenter adjusted the effective display size of the images until the participant was unable to identify the image. The effective display size was adjusted by changing the viewing distance, by toggling the onscreen image size, or both. We did this both ways: gradually moving out from a near viewing location and/or decreasing the onscreen image size and gradually moving from a far viewing location to get closer to the monitor and/or increasing the onscreen image size. We repeated this procedure until the effective display size threshold, at which participants were just unable to identify the events, was found. See Table 3 for each individual participant's effective display size. Then participants were seated at this “just being unable to identify” distance, and the seat was adjusted so that their eyes were aligned with the center of the screen, and they completed the first four conditions of the perception experiment. 

Figure 4

 

(a) Examples of black-and-white images used to find the identification thresholds for individual amblyopic participants. There were 80 similar images of common daily settings. (b) Examples of events used in the perceptual experiment. The images were unblurred. We have sought and received permissions for use of the images from individuals whose images were identifiable in the experimental stimuli.

Table 3

 

Effective display size (in DVAs) for amblyopic participants in Experiment 3. Notes: All four control participants were tested with viewing distance of 330 cm and onscreen display size of 13.5 cm × 9 cm (the smallest effective display size of the clinical group). DVA = degree visual angle.

 

In the perception experiment, amblyopic participants attempted to identify daily events (same events as in Experiment 2) from unblurred black-and-white images with individualized effective display sizes. Otherwise, the experimental task, conditions, and orders were identical to those in Experiments 1 and 2. During the first visit, participants performed Conditions 1 to 4. Then, they revisited the laboratory after five days to complete a free recall and an identification condition (Condition 5). 

Four control participants were recruited after all the amblyopic participants completed the study. Control participants followed the identical protocol and completed the five conditions of this experiment using the smallest effective display size used by amblyopic patients (i.e., with viewing distance of 330 cm and onscreen display size of 13.5 cm × 9 cm). 

Data processing

There were 12 participants, who completed 1,056 trials altogether. Two raters independently coded event descriptions (using the same coding standard as in Experiments 1 and 2), and their coding was the same in 981 of 1,056 trials, making the interrater reliability 93% (Uebersax, 1987). In trials in which the raters coded differently, one rater's coding was randomly picked. 

Results

In Experiment 3, we studied eight amblyopic participants and four normally sighted participants. There was no significant age difference between the two groups, t(7.76) = −0.34, p = 0.74. The vision tests revealed that the amblyopic participants had lower binocular VA, t(7.61) = 6.65, p < 0.001, and marginally lower CS, t(10) = −2.21, p = 0.052. The group average mean visual field defect of the amblyopic observers was no different from that of the controls, t(9.88) = −2.10, p = 0.06. None of the participants had any scotomas. See Table 4. 

Table 4

 

Summary of participants' vision data, Experiment 3. Notes: Results listed as mean ± 1 SD.

 

In the perception test, we investigated if adding motion allowed participants to identify events that were not identifiable with static images alone. For the amblyopic patients, a repeated-measures ANOVA revealed that event identification performance was affected by condition, F(4, 28) = 24.2, p < 0.001, partial η² = 0.78, with a higher rate of recall in the postmotion Conditions 4 and 5 than in the premotion Conditions 1 and 2. See Figure 5. Performance was not affected by event, F(7, 49) = 1.95, p = 0.081, or event-condition interaction, F(28, 196) = 0.694, p = 0.87. All of the controls identified the events correctly in all conditions. This task was not challenging for them, because of their normal vision. We hence focus on how motion aided amblyopic patients to identify events. 

Figure 5

 

Performance for the amblyopic and normally sighted participants in five conditions. Error bar = 1 SE. Visual stimuli used in each trial: Condition 1: single static frame of clear image; Condition 2: 20 frames of clear images with motion masks; Condition 3: 20 frames of clear images with motion; Condtions 4 and 5: single static frame of clear image.

Despite using different visual stimuli, the trends were similar to those in Experiments 1 and 2. In Condition 1, events were perceived correctly in only 10 of 192 trials, or 5.2% (95% CI = [2.1%, 8.4%]). In Condition 2, where sequences of images were available, performance did not improve. Participants still identified only 10 events (but the proportion increased to 15.6% (95% CI = [6.6%, 24.6%]) because the total number of trials decreased to 64). So, regardless of the number of images, events were effectively imperceptible in Conditions 1 and 2. 

When both static image information and motion information were available in condition 3, 36 events (of 64) were identified among all participants (or 56.3%, 95% CI = [44%, 68.5%]). Comparing performance in Condition 2 and in Condition 3, the added motion information yielded a large and significant improvement in event perception, F(1, 7) = 21.51, p = 0.002. The good performance was retained in the postmotion Condition 4, with the same images as in Condition 1. The rate of identification was 43.8% (84 of 192 trials, 95% CI = [36.7%, 50.8%]), significantly better than in Condition 1, F(1, 7) = 34.85, p < 0.001. 

During the second visit, amblyopic participants first performed a free recall, recalling 23 of 64 events. Then, they were shown the same visual stimuli with previously measured individualized effective display size and identified the events from clear static images. The rate of correct identification in Condition 5 was 42.2% (81 of 192 trials, 95% CI = [35.2%, 49.2%]). Performance was significantly better than in Condition 1, F(1, 7) = 49.63, p < 0.001, and was not worse than in Condition 4, F(1, 7) = 0.094, p = 0.77, which was tested during the first session immediately after the motion condition. Performance in Condition 5 depended on whether the events were successfully identified in Condition 3 (with motion) but not whether events were recalled. For events that were identified in Condition 3, when the corresponding static images appeared again in Condition 5, 67.6% (73 of 108 trials) of these static images were reidentified. For events that were not recalled, when the corresponding images appeared in Condition 5, in 43 of 123 trials (or 35.0%), the events were accurately identified. For events that were recalled, when the corresponding images appeared in Condition 5, in 31 of 69 trials (63.2%), the events were not identified. This suggested that free recall performance and identification in Condition 5 were rather unrelated. 

General discussion

We studied how optical flow information might allow events in everyday life to be perceived despite poor image-based information. Specifically, optic flow interacts with images and calibrates them so they become spatiotemporally meaningful (i.e., perceptible as events). When the optic flow stops, the calibrated image structure remains. Despite poor image quality, calibrated image structure information then acts as embodied memory that preserves events previously specified by optic flow. This embodied memory engenders accurate and stable perception of events with ongoing motion and after motion stops. In this process, high image resolution is not required, and thus optic flow and embodied memory are expected to aid observers with blurry vision to perceive events accurately and stably. 

Our results supported this theoretical proposal. In Experiments 1 and 2, with participants of different vision loss, we showed that motion-generated optic flow information compensated for the lack of higher spatial frequencies in visual stimuli and enabled effective event perception for AMD patients, amblyopic patients, and normal controls. In Experiment 3, using a slightly different experimental design, we showed that when the clear images were too small for amblyopic observers to perceive, adding motion helped. Participants benefited from the optic flow–image structure interaction and successfully identified events in the motion and postmotion Conditions 3, 4, and 5. In the first two experiments, high spatial frequencies were removed from the visual stimuli as well as from the observing eyes for the clinical groups because of vision loss. This is like adding one filter to another similar filter, and the final outcome was not affected by the additional, similar filter. Hence, in Experiments 1 and 2, there was no difference in behavior between the clinical and the control groups. In Experiment 3, there was no blur in the stimulus images. High spatial frequencies existed in the stimuli and were available to normally sighted observers, but the high spatial frequencies were not available to the amblyopic observers because of their vision loss. Hence, there was a clear distinction in performance between the clinical and the control groups in Experiment 3. 

Despite the differences in participant types, vision status, and experimental materials, the trend of performance for all groups of participants was similar. With only static blurry images in Conditions 1 and 2, event perception was poor. In Condition 3, images were played in sequence so continuous motion was detectable and visual events became perceptible. Moreover, motion-based information calibrated the blurry images to yield a higher rate of identification in the postmotion Condition 4 than in the premotion condition 1. As previously shown, this improvement was not merely due to having observed the blurry images repeatedly (Pan & Bingham, 2013). Although performance in Condition 4 declined from that in Condition 3, performance maintained from Condition 4 to Condition 5, which was tested five days later. The performance drop from Condition 3 to 4 was because one important source of information, namely, optic flow, was removed. Although image structure preserved information in optic flow and enabled good performance postmotion, event perception was still the most robust when both dynamic and static information was available in real time. On the other hand, a high rate of identification persisted from Condition 4 to Condition 5, and this supports the hypothesis that embodied memory (formed when optic flow and image structure interact) allowed the calibration of the blurry images to last and events specified by optic flow five days ago to be still perceptible. This is because previously perceived events were preserved in calibrated blurry image structure (equivalently, in embodied memory). Hence, upon seeing the blurry images again, participants extracted events from embodied memory to accomplish the identification task. This is different from the traditional “memory-in-the-head,” where participants remembered the events for five days (as in storing the perceived events in their head), because when asked what events they had seen from last time in the laboratory, their recall rates were lower than their identification performance in Condition 5. Furthermore, contrasting the free recall rates between the older healthy control participants in Experiment 1 and younger healthy control participants in Experiment 2, recall performance was poorer for the older participants (older controls in Experiment 1: M_recall = 0.30, SD_recall = 0.46; younger controls in Experiment 2: M_recall = 0.45, SD_recall = 0.50), F(1, 126) = 3.67, p = 0.06. Characteristic of human memory-in-the-head, free recall should decline with age, as a result of normal aging (Cavanaugh, 1993; Nilsson, 2003). However, for both older and younger participants, their performance in Condition 4 was equivalent to that in Condition 5. This implied that the mechanism involved in completing the task in Conditions 4 and 5 must not be the mechanism responsible for free recall, namely, memory-in-the-head. Instead, it is embodied memory, and embodied memory did not decline with age. 

The implication of these results is twofold. First, if considering visual tasks and functions in typical daily environments, events are more prevalent than static objects as visual targets. So, when assessing visual function, it is worthwhile to test how events, in addition to static stimuli such as letters or colored shapes, are perceived. Whether events are perceptible is not determined by the quality of static image-based information alone but also by motion-based information. Second, static image–based information and motion-based information facilitate each other to yield effective and stable event perception. Motion calibrates image structures and assigns spatial meanings to them. Detecting continuous optic flow with impaired static vision, discrete blurry images can be linked to one another to form comprehensible events. Optic flow compensates for the loss of static visual details. Moreover, calibrated images preserve the spatial meanings once specified by optic flow, making events continue to be perceptible in an enduring way once motion is no longer available. This role of preservation does not require static images to contain high spatial frequencies. Thus, when viewing blurry events (in Experiments 1 and 2), the integration of optic flow and image structure information allowed observers with poor static vision to perceive events as effectively (in terms of accuracy and persistency of perception) as their normally sighted counterparts. 

Comparing across experiments, there was no difference in the trend of performance between the various clinical groups. Vision loss of the AMD observers (when viewing with both eyes) was mild (≈ 20/40), and vision loss for the amblyopic observers (when viewing with the amblyopic eye) was moderate (≈ 20/80 in Experiment 2 and ≈ 20/130 in Experiment 3). Other evidence suggests that more severe loss (>20/200) might nevertheless yield good performance in our task (West et al., 2002). Therefore, so long as an observer can process motion reliably and have access to some static image information, they should be able to take advantage of the powerful combined optic flow–image structure system to perceive events in real time as well as after postmotion. 

Comparing younger and older participants' performance in this study, we noticed that there was a bigger drop of performance from Condition 3 to Condition 4 for older observers. The AMD patients and the age-matched controls (average ages being 77.54 and 78.18 years) in Experiment 1 identified 82% of events in Condition 3 and 58% of events in Condition 4. The amblyopic patients and their age-matched controls (average ages being 19.38 and 20.12 years) in Experiment 2 identified 69.5% events in Condition 3 and 63.5% in Condition 4. The amblyopic patients (average ages being 17.38) in Experiment 3 identified 56.3% events in Condition 3 and 43.8% in Condition 4.² Pan and Bingham (2013) used a similar design to test healthy participants aged 19 to 55 (average age 31.1 years). These adult participants identified 88% events with ongoing motion in Condition 3 and recognized 77% events from blurry static images in Condition 4. The drop in performance from Conditions 3 to 4 was more than twice as large for older participants (i.e., the AMD's and the age-matched controls in Experiment 1). Although the ability to recognize familiar objects decreases with age, especially over age 80 (Dangnelie, 2013), we did not think this bigger drop in performance was because of the reduced recognition ability because if this were true, then older observers should exhibit a bigger drop of performance from Condition 4 to Condition 5 as compared with younger observers. This was not the case. The drop of performance was equivalent for all age groups between Conditions 4 and 5. Thus, the fact that older observers did well in Condition 3 but less well in Condition 4 might suggest that in these participants, blurry image structure failed to preserve information in optic flow as effectively. There appears to be an age difference in integrating optic flow and image structure information. Our data imply that as age increased, this ability might decrease. See Table 5. 

Table 5

 

Comparing the rate of event identification of observers of different ages.

 

Conclusion

In three experiments, we showed that motion-generated optic flow information aided event perception from images that were too blurry to support perception based on static visual information alone. Observers with mild or moderate vision loss accurately identified events with ongoing motion (and therefore continuous optic flow) and successfully identified events from static images postmotion. There was no behavioral difference in performance between observers with different severity of vision loss. This is because observers were able to integrate optic flow and image structure information and use them to perceive blurry events in an accurate and stable way. 

These results might explain why individuals with severely low VA (20/200 or worse) successfully performed, in unfamiliar environments, daily tasks that entail recognition of common objects and then guide their actions relative to them (Leat et al., 1999). The implications for vision care and rehabilitation are that, first, event perception should be tested as a measure of visual function and, second, intervention methods should aim to improve patients' ability to generate strong optic flow by moving actively and efficiently and to combine optic flow and image structure when performing various activities of daily living. 

Acknowledgment

This project was funded by the National Natural Science Foundation of China (Grant 31571116) and Sun Yat-sen University (Grant 16wkpy29), awarded to the first author. The funding agencies had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. 

Commercial relationships: none. 

Corresponding author: Jing Samantha Pan. 

Email: [email protected]. 

Address: Department of Psychology, Sun Yat-sen University, Guangzhou, China. 

Footnotes

¹  The equipment available at the Zhongshan Ophthalmic Centre was for clinical use, instead of research, and measures only the central 30° of visual field. Because participants were allowed to move their eyes while viewing, this measurement should be acceptable although different from the VF measurement in Experiment 1.

Footnotes

²  The healthy controls in Experiment 3 identified all events in all conditions from the unblurred images. So they were excluded from this discussion.