Pupil responses to high-level image content

Abstract

Abstract

Abstract:

Abstract  The link between arousal and pupil dilation is well studied, but it is less known that other cognitive processes can trigger pupil responses. Here we present evidence that pupil responses can be induced by high-level scene processing, independent of changes in low-level features or arousal. In Experiment 1, we recorded changes in pupil diameter of observers while they viewed a variety of natural scenes with or without a sun that were presented either upright or inverted. Image inversion had the strongest effect on the pupil responses. The pupil constricted more to the onset of upright images as compared to inverted images. Furthermore, the amplitudes of pupil constrictions to viewing images containing a sun were larger relative to control images. In Experiment 2, we presented cartoon versions of upright and inverted pictures that included either a sun or a moon. The image backgrounds were kept identical across conditions. Similar to Experiment 1, upright images triggered pupil constrictions with larger amplitudes than inverted images and images of the sun evoked greater pupil contraction than images of the moon. We suggest that the modulations of pupil responses were due to higher-level interpretations of image content.

Introduction

Pupils respond to light but also to internal mental states. For example, when people are aroused or have a negative experience, their pupil dilates (e.g., Bradley, Miccoli, Escrig, & Lang, 2008; Hess, 1975). Exploratory behavior during demanding tasks (Beatty & Wagoner, 1978; Bradshaw, 1967; Kahneman & Beatty, 1966) and decision-making (Einhäuser, Koch, & Carter, 2010) are also typical arousal-related processes that affect pupil size. Pupil dilation is well studied and often linked to sympathetic activations (Aston-Jones & Cohen, 2005; Gilzenrat, Nieuwenhuis, Jepma, & Cohen, 2010; Jepma & Nieuwenhuis, 2011; Loewenfeld & Lowenstein, 1993; Rajkowski, Kubiak, & Aston-Jones, 1993). There are, however, several indications that pupil response modulations can be attributed to cognitive changes other than sympathetic arousal (Steinhauer, Siegle, Condray, & Pless, 2004). For instance, the amplitude of pupil constrictions is attenuated when ocular dominance of an eye is decreased (Báràny & Halidén, 1948; Brenner, Charles, & Flynn, 1969; Lowe & Ogle, 1966; Richards, 1966), and pupil size adjusts to perceptual changes in brightness and contrast during binocular rivalry (Fahle, Stemmler, & Spang, 2011; Naber, Frässle, & Einhäuser, 2011) and perceptual brightness illusions (Laeng & Endestad, 2012). Pupil constrictions are also induced by a change in other low-level features such as color or motion (Barbur, Harlow, & Sahraie, 1992; Kohn & Clynes, 1969). The question remains whether these types of responses are low-level driven or consequences from higher-level processes. Here we address this by disentangling high-level effects on pupil diameter from low-level effects. Observers are shown pictures with or without a sun that are presented upright or inverted while matched for image features across conditions. In two experiments, we demonstrate that upright images induce pupil constrictions with larger amplitudes than inverted images. We additionally show that the decrease in pupil diameter as a response to the onset of upright images with a sun is larger as compared to the onset of upright images without a sun. 

Methods

Observers

Twenty-six observers (including one author) participated in the first experiment and a new group of 12 observers participated in the second experiment. All participants had normal or corrected-to-normal vision, were naïve to the purpose of the experiment (except for the author), and gave informed written consent before the experiment. The experiments conformed to the ethical principles of the Declaration of Helsinki and were approved by the local ethics commission of Harvard. 

Stimuli

Experiment 1

To manipulate high-level brightness concepts of objects independent of their physical brightness, we used 20 distinct digital images of scenes that contained a sun and 20 distinct images without a sun while we controlled for luminance and contrast (see http://www.marnixnaber.com/sunnyimages). Scenes with a sun may have appeared to be bright (Figure 1a) but the actual luminance of the digital suns was not brighter than the background of a white page. Nonetheless, we wanted to fully ensure that luminance or contrast could not confound the brightness of images that included a sun. Overall luminance and contrast of sun images were therefore lower (20.45 cd/m² ± 1.04) than control images (25.05 cd/m² ± 0.93; luminance: t(19) = 22.93, p < 0.001; contrast: t(19) = 3.07, p < 0.01). We controlled for other image statistics by pairing each sun image with a control image without a sun that was approximately matched in scene layout, color, and content. All images were 25.88° × 19.41° (landscape, 1280 × 960 pixels) in size and shown at the center of the screen with a gray background of 30.99 cd/m². Images were taken from the web (images.google.com). 

Figure 1

 

Pupil responses during the viewing of natural scenes. (a) Examples of sun and control images (columns), and upright and inverted images (rows). These images serve as an example and were not used in the actual experiment. The images were made available by Paul Sullivan (http://www.flickr.com/photos/pfsullivan_1056/4626582733/in/photostream/) and Rajeev Nair (http://www.flickr.com/photos/rajeevnair1981/377595734/) under the creative commons license. (b) Average transient changes in pupil diameter (baseline normalized to image onset; solid lines) across observers in response to blank (α; higher luminance) and presented images (β; lower luminance). Note the sharp and transient constriction to the blank onset (α) is a result of an increase in luminance. The second deflection (β) was not a result of luminance but a response to image onset. The height of the timeline on top of the Figure indicates the average luminance of each stimulus (i.e., image of preceding trial [n–1], blank, and image of current trial [n]). (c) Average change in pupil diameter between 1 and 3 s after image onset (see shaded area in (b); * p < 0.05; ** p < 0.01; *** p < 0.001).

Experiment 2

The stimuli in the second experiment consisted of 10 cartoon images with a sun and 10 images with a moon (see http://www.marnixnaber.com/sunnyimages). To further control for the effects of low-level image features on pupil constrictions, each image pair's background scene was identical. The sun cartoon images had an average luminance of 20.22 cd/m² ± 8.22 and moon cartoon images 20.07 cd/m² ± 8.11, and both image sets were equal in luminance, t(9) = 1.00, p = 0.343. All other stimulus aspects were similar to Experiment 1. 

Apparatus

Stimuli were presented on a 21-inch Samsung Syncmaster CRT screen (Samsung, Samsung Town, Seoul, South Korea) at a fixed viewing distance of 70 cm. Observers' heads were supported by a chin- and forehead-rest. The refresh rate of the screen was 85 Hz and the resolution was 1600 × 1200 pixels. Stimuli were generated on a Dell computer (Dell, Round Rock, TX), using Matlab (Mathworks, Natick, MA). Pupil diameter of one eye was tracked with an infrared sensitive camera (EyeLink 1000, SR Research, Osgoode, Ontario, Canada) at a rate of 1000 Hz. 

Procedure

Experiment 1

Observers were shown a total of 400 images, divided in five separate blocks. Each displayed image was either with or without a sun, and the image was either upright or inverted (2 × 2 design). Images were inverted to distort the subjective global impression of the scenes while image statistics were kept identical. All four conditions were randomly intermixed. Images were shown for 3 seconds and a blank gray screen (luminance: 30.99 cd/m²) was presented between images presentation with a duration of 0.61 s (SD = 0.044). As the blank screens had a higher luminance than the presented images (see Stimuli), the blank onsets induced pupil constrictions. To keep the task as “natural” as possible, observers were allowed to free-view the images and were explicitly instructed to carefully inspect each image in detail. Observers could take breaks between blocks and the eye-tracker was calibrated at the start of each block with a 13-point calibration grid. 

Experiment 2

All procedural aspects of Experiment 2 were similar to Experiment 1 except for the number of presented images, interstimulus intervals, and fixation. Observers were shown a total of 200 images, divided in five separate blocks. To prevent residual effects of preceding trials, Experiment 2 had longer and variable blank screen interval durations between the image presentations (M = 2.70 s, SD = 0.57). A fixation point was shown during the blank intervals between images to prevent effects of gaze position on pupil diameter before image onset. 

Data analysis

The Eyelink system outputs an arbitrary unit rather than absolute pupil diameter that depends on the distance between camera and eyes, and the infrared sensitive detection thresholds. We could nonetheless estimate average pupil diameter across observers at approximately 4.6 mm (SD = 0.3). To compare results across observers independent of these variable factors and relative to the image onsets, we normalized pupil traces by subtracting pupil diameter at image onset. We further controlled for the effects of the eye's visual angle on pupil size. The average change in pupil diameter was calculated in the period after the responses (t = 1 s) until image offset (t = 3 s). The main effect of image inversion on average pupil diameter in millimeters (i.e., the average differences in pupil diameter between upright and inverted conditions) was based on the mean across both sun and control conditions. Differences in pupil diameter between sun and control conditions were calculated only for the upright condition (i.e., when global scenes had a natural, undistorted viewpoint). 

Image luminance was calculated by averaging the luminance of all the image pixels in the display. Image contrast was based on the variance across all luminance values of the pixels. 

Results

Experiment 1—Pupil responses to natural scenes

The pupil responses to blank and image onsets consisted of temporary constrictions that were modulated by image content (Figure 1a, b). As described earlier, images were presented in sequence (3 s each) and blank screens (∼0.6 s) were shown between images (see timeline in Figure 1b). A blank screen had a higher luminance than the preceding image (see Methods), hence the blank onset led to a typical luminance induced transient constriction followed by a recovery to baseline (α). The subsequent pupillary constriction (β) was a result of the following image onset. Merely based on differences in luminance, pupil dilations were to be expected (sun: 0.25 mm; cartoon: 0.13 mm; field diameter: 22.6, age: 25 years; Watson & Yellott, 2012). Yet, as mentioned in our Introduction, the onset of changes in contrast rather than luminance also have been shown to elicit constrictions (Naber et al., 2011). 

In line with these recent results, our stimuli also evoked pupil constrictions (see β in Figure 1b). These constrictions were responses to features other than brightness because the image luminance was lower than the preceding blank. The high-level content in the images was also a modulator of these pupil responses. Upright images induced larger constrictions in pupil diameter than inverted images (Figure 1c; for statistics, see Table 1). Images that depicted a sun also led to larger pupil constrictions and a smaller average pupil diameter than images without a sun. The “inversion effect” and “sun effect” were approximately 0.13 mm (SD = 0.09) and 0.07 mm (SD = 0.05) in diameter, respectively. The inversion effect on pupil diameter was larger than the sun effect, t(25) = 2.70, p = 0.012. The pupil response to the preceding blank screen (α) could not account for these effects as there were no baseline differences between the upright sun and control conditions, t(25) = 1.25, p = 0.221, nor between the upright sun and inverted sun conditions, t(25) = 0.08, p = 0.934, for the raw unnormalized pupil size measures at image onset. Hence, preceding trials did not affect the inversion and sun effect. As such, we conclude that the processing of image features other than brightness caused pupil constrictions and that the high-level interpretation of image content modulated these responses. 

Table 1

 

T tests on average pupil diameter per natural scene image conditions.

Image conditions (n = 26)	Upright sun	Upright control	Inverted Sun
Upright control	t = 2.23, p = 0.035
Inverted sun	t = 5.39, p < 0.001	t = 5.49, p < 0.001
Inverted control	t = 3.67, p = 0.001	t = 3.84, p < 0.001	t = 1.42, p = 0.169

Experiment 1—Gaze control

The effects outlined above were independent of changes in low-level image statistics because the luminance and contrast was controlled for (see Methods). Nonetheless, observers may have fixated more often on brighter image areas in the upright and sun conditions, resulting in increased luminance levels at foveal retinal regions and therefore stronger pupil constrictions. To assess whether gaze behavior confounded the effect of sun and image inversion on pupil diameter, we computed density heat maps of gaze fixations in screen coordinates across conditions. As shown in Appendix Figure 1a, fixations were biased around image centers for all conditions. Average fixation eccentricity slightly differed across conditions (p < 0.05; see Appendix Figure 1b) in which sun images and upright images received more peripheral fixations than control images and inverted images, respectively. These differences were, however, small (<0.5°) and it is therefore unlikely that these fixations resulted in increased luminance or contrast levels in the fovea for sun and upright image conditions. Yet, to further control for potential confounds of fixations, we computed the luminance and contrast levels of fixated image regions per condition. Data showed that fixated regions of images including a sun were less bright (Appendix Figure 1c; p < 0.05 for aperture radius > 1.90°) and had a lower contrast (Appendix Figure 1d; p < 0.05 for aperture radius > 0.01°) than those in control images, and these differences remained significant across all fixation times during image presentation (p < 0.05). Image luminance at fixated regions was not significantly different between inverted and upright conditions, neither for sun nor for control images (p > 0.05). Image contrast was also not significantly different between upright and inverted conditions for sun images. Contrast of fixated regions of control images only differed between upright and inverted conditions at large apertures (p < 0.05 for aperture radius > 3.60°). Thus, gaze behavior and variations in image luminance or contrast were unlikely to confound the differences in pupil responses across image conditions. 

Experiment 2—Pupil responses to cartoon images

To control for more complex image features, we conducted a second experiment in which a new group of 12 observers were shown artificial cartoon scenes that contained identical image statistics in both the sun and control condition (Figure 2a; for full image set, see http://www.marnixnaber.com/sunnyimages). Similar to Experiment 1, images had a lower luminance than the preceding blank screen (see Methods). In contrast to an expected pupil dilation due to a decrease in screen luminance (0.20 mm; field diameter: 22.6, age: 25; Watson & Yellott, 2012), an upright image onset triggered a pupil constriction. The differences in pupil diameter across conditions (Figure 2b) were qualitatively comparable to Experiment 1 (see Figure 1b). The onset of upright images resulted in pupil responses with larger amplitudes than inverted images (Figure 2c; for statistics, see Table 2). The pupil also significantly changed to smaller diameters for upright sun images as compared to upright moon images. The effects of image inversion and the presentation of a sun on average pupil diameter was 1.12 mm (SD = 0.73) and 0.35 mm (SD = 0.50), respectively. The inversion effect was significantly larger than the sun effect, t(11) = 3.74, p = 0.003. There were neither pupil diameter baseline differences at image onset between upright sun and control images, t(11) = 0.64, p = 0.535, nor between upright sun and inverted sun images, t(11) = 0.37, p = 0.726. As with the data of Experiment 1, we controlled for gaze by analyzing the average gaze fixation eccentricity, and found no differences in eccentricity between upright sun and upright control images, t(11) = 0.78, p = 0.454, nor between upright sun and inverted sun images, t(11) = 1.56, p = 0.147. Furthermore, luminance and contrast around fixated image regions did not differ across conditions (p > 0.05). Hence, we conclude that the pupil was smaller during the viewing of upright scenes as compared to inverted scenes, and smaller during the viewing of upright sun images than upright control images. 

Figure 2

 

Pupil responses to artificial images. (a) Examples of presented artificial sun and control images (columns), and upright and inverted images (rows). (b) Average change in pupil diameter (solid lines) as a function of time around image onset per image and inversion condition across observers. (c) Average change in pupil diameter between 1 and 3 s after image onset.

Table 2

 

T tests on average pupil diameter per cartoon image conditions.

Image conditions (n = 13)	Upright sun	Upright control	Inverted sun
Upright control	t = 2.52, p = 0.029
Inverted sun	t = 5.32, p < 0.001	t = 3.73, p = 0.003
Inverted control	t = 6.92, p < 0.001	t = 3.27, p = 0.008	t = 0.21, p = 0.837

Discussion

We presented images of natural and artificial scenes to observers while we measured their pupil responses triggered by image onsets. We demonstrated that the onset of upright images resulted in constrictions of an observer's pupils. The viewing of inverted images, however, resulted in smaller (Experiment 1) to almost absent (Experiment 2) constriction amplitudes. Furthermore, the onset of images that included a sun induced pupil constrictions with larger amplitudes than images without a sun. The pupil responses and the differences in pupil diameter across these image conditions must have resulted from mechanisms other than low-level image processing because we controlled for image luminance, image contrast, and gaze across conditions. We suggest that the high-level visual processing of scenes underlie these findings because of the following observations: First, inverted sun images had identical image statistics as the upright sun images but induced attenuated pupil constrictions (inversion effect). Second, images with a sun had a lower luminance and contrast than images without a sun in Experiment 1 but they triggered larger constriction amplitudes (sun effect). Third, the luminance of the blank screens between image presentations was higher than the actual images. This difference in luminance predicted that image onsets would trigger pupil dilations, but we observed the opposite pattern. These phenomena can thus not be accounted for by low-level image features. We propose that pupillary responses can be evoked by the processing of the abstract content of images, such as the valence or interpretation associated with particular objects or conditions (e.g., high light levels that may damage the retina). Pupil responses are primarily driven by subcortical, arousal, or low-level features, but as image content is analyzed by late stages of cortical visual processing, the sun and inversion effect point to a high-level influence of scene perception and object processing. This study thus provides evidence that, in addition to arousal-related pupil dilations, higher-level processes can affect pupil responses. 

The question remains what type of higher-order cognitive process underlies pupil responses. Image inversion had the largest effect on the pupil responses. Image inversion is known to impair recognition performance of stimuli such as pictures of faces, buildings, and cartoons (e.g., Goldstein, 1965; Scapinello & Yarmey, 1970; Strother et al., 2011; Valentine & Bruce, 1986; Van Belle, De Graef, Verfaillie, Rossion, & Lefèvre, 2010; Yin, 1969). Studies on face recognition suggest that figure inversion distorts the holistic and configural processing of features (e.g., Farah, Tanaka, & Drain, 1995; Rossion & Gauthier, 2002; Tanaka & Farah, 1993). Thus, inversion impairs complex relations between features that are necessary to recognize a stimulus. As complex and global features are generally processed at later stages in the visual hierarchy (Felleman & Van Essen, 1991; Kobatake & Tanaka, 1994; Serre, Oliva, & Poggio, 2007), this implies that high-level processing is impaired by figure inversion. In line with this, an inverted face decreases activity in fusiform face area (FFA), a higher-order brain area (Kanwisher, Tong, & Nakayama, 1998). Hence, it is not unlikely that the attenuated pupil constrictions to inverted scenes are a result of disrupted high-level processing of complex image features. 

The subjective perception of increased brightness for images with a sun could be a potential explanation for the sun effect. Natural images with a sun appear to be bright and the cognitive mechanisms that process such low-level features may have feedback connections to the autonomic nervous system that controls pupil size (note that the pupil constrictions to sunny computer images are much smaller than pupil constrictions to real-life sunny scenes). A recent study indeed proposed that the subjective illusion of seeing a bright stimulus can constrict the pupil (Laeng & Endestad, 2012). There are, however, several reasons why these results are likely to be distinct from ours. For instance, some of the brightness illusions were confounded by local contrast, a feature that can induce pupil constrictions as well (Naber et al., 2011). The reported pupil traces by Laeng and Endestad (2012) also depicted transient constrictions at remarkably high speeds (more than ∼100 mm/s) exactly at image onset, a phenomenon that deviates from standard pupil responses to sequences of images (e.g., Naber, Hilger, & Einhäuser, 2012; Qin, Hermans, van Marle, & Fernández, 2012). Lastly, the illusion that induced the strongest relative pupil constriction in Laeng and Endestad's data closely resembled a sun (see the Asahi illusion in figure 1A and 1B; 2012). This hints at the involvement of mechanisms other than brightness perception. Moreover, brightness perception cannot explain why the viewing of upright images induces greater pupil contraction than inverted images. 

Another interpretation for our findings is that the upright control (i.e., without a sun) and inverted images are associated with pronounced pupil dilations because of increased arousal and effort. Fluctuations in pupil size are typically attributed to changes in arousal and the connection between pupil dilation and cognitive effort is well-established (e.g., Loewenfeld & Lowenstein, 1993). This proposition, however, implies that the processing of inverted scenes and images without a sun needs more effort, is more arousing, or draws more attention than the processing of upright scenes and images with a sun. It further suggests that an evoked pupil constriction, as a response to an upright image onset, is the result of a sudden decrease in arousal. This seems rather unlikely as it conflicts with recent findings, showing that increased difficulty in the detection and identification of natural images results in increased pupillary constriction, not dilation (Naber et al., 2012). A much simpler explanation for our findings is the effect of visual processing on pupil constrictions (Barbur, 1995; Barbur, Wolf, & Lennie, 1998). We deem it more likely that the increased visual processing of images results in stronger pupil constrictions. Within the context of the current paper, this implies that images with a sun attract more attention and increase visual processing because a sun has a potentially high light level that could damage the retina. In addition, upright images attract more attention and processing resources than inverted images because inverted images display less relevant information and there is thus less to process. Indeed, inverted stimuli activate weaker neural responses than upright stimuli in brain areas that are important for the processing of complex features (Gauthier & Tarr, 2002; Rossion & Gauthier, 2002). Also congruent with the propositions above, artificial scenes are processed slower (Rousselet, Joubert, & Fabre-Thorpe, 2005) and we observed a delayed effect of the sun on pupil responses for artificial cartoon images. Lastly, the facilitated encoding of a stimulus (Naber, Frässle, Rutishauser, & Einhäuser, 2013) and increased levels of attention (Binda, Pereverzeva, & Murray, 2013) evoke pupil constrictions with larger amplitudes. Hence, we find it tempting to suggest that the visual processing of images and the activation of abstract representations of potentially harmful objects can directly affect the central nervous system (CNS) and pupil size through a process that is unrelated to sympathetically activated states of arousal. The cognitive processing of images may thus have a greater impact on basic reflexes and other CNS-driven physiological functions than previously assumed. 

Acknowledgments

We thank Jon Cant and Marius ‘t Hart for their experimental suggestions. This work was supported by the Dutch NWO Rubicon grant (446-11-013), cofunded by the Marie Curie Cofund Action. 

Commercial relationships: none. 

Corresponding author: Marnix Naber. 

Email: [e-mail]. 

Address: Vision Sciences Laboratory, Department of Psychology, Harvard University, Cambridge, MA, USA. 

Appendix

Figure A1

 

Fixation control. (a) Fixation density maps aligned to image coordinates per image condition. The density maps were created by adding a circular-shaped Gabor of ∼ 1.2° at each fixated location per observer. Fixation maps were then normalized between zero and one per observer and then averaged across observers. (b) Average gaze fixation eccentricity from image center per condition. (c) Average image luminance and (d) contrast at fixation as a function of image aperture size (i.e., the radius of the selected window in the image around fixation) around fixation. Transparent patches around average indicate the standard error around the mean across fixations. The black patches indicate at which fixation apertures image luminance or contrast significantly differed between the sun and control conditions (p < 0.05). The dark blue patch indicates at which apertures features differed between upright and inverted images for the control condition (there were no significant differences for the sun condition).