Pupillary response to representations of light in paintings

Abstract

It is known that, although the level of light is the primary determinant of pupil size, cognitive factors can also affect pupil diameter. It has been demonstrated that photographs of the sun produce pupil constriction independently of their luminance and other low-level features, suggesting that high-level visual processing may also modulate pupil response.

Here, we measure pupil response to artistic paintings of the sun, moon, or containing a uniform lighting, that, being mediated by the artist’s interpretation of reality and his technical rendering, require an even higher level of interpretation compared with photographs. We also study how chromatic content and spatial layout affect the results by presenting grey-scale and inverted versions of each painting. Finally, we assess directly with a categorization test how subjective image interpretation affects pupil response.

We find that paintings with the sun elicit a smaller pupil size than paintings with the moon, or paintings containing no visible light source. The effect produced by sun paintings is reduced by disrupting contextual information, such as by removing color or manipulating the relations between paintings features that make more difficult to identify the source of light. Finally, and more importantly, pupil diameter changes according to observers’ interpretation of the scene represented in the same stimulus.

In conclusion, results show that the subcortical pupillary response to light is modulated by subjective interpretation of luminous objects, suggesting the involvement of cortical systems in charge of cognitive processes, such as attention, object recognition, familiarity, memory, and imagination.

Introduction

The pupil is the central opening of the iris that regulates the intensity of light entering the eye to adjust retinal illumination and optimize vision (Loewenfeld, 1993). Light increments produce pupillary constriction (miosis), whereas light decrements produce pupillary dilation (mydriasis). This is known as pupillary light reflex (PLR), which is controlled by the autonomic nervous system (Gamlin & Clarke, 1995; Loewenfeld, 1993). Currently, a consistent body of evidence demonstrates that the PLR is not merely a basic low-level mechanism, showing that, even if the intensity of light is the primary determinant of the pupil size, non-visual factors can also affect the pupil diameter. 

First studies of pupillometry showed that the pupil dilates not only in the dark but also in response to an increase in level of arousal (Bradley, Miccoli, Escrig, & Lang, 2008; Henderson, Bradley, & Lang, 2014; Hess & Polt, 1960; Snowden, O'farrell, Burley, Erichsen, Newton, & Gray, 2016), associated with an increased sympathetic activity (Aston-Jones & Cohen, 2005). Other studies demonstrated that the pupil dilates during the execution of mental tasks that require cognitive load (Beatty, 1982; Hess & Polt, 1964; Just & Carpenter, 1993), memory effort (Beatty & Kahneman, 1966; Goldinger & Papesh, 2012; Kafkas & Montaldi, 2011; Kahneman & Beatty, 1966; Papesh, Goldinger, & Hout, 2012), and decision making processes (De Gee, Knapen, & Donner, 2014; Einhäuser, Stout, Koch, & Carter, 2008). 

More recent studies found that the pupil response can be modulated by high-level visual processes, such as attention (Binda & Murray 2015a; Binda, Pereverzeva, & Murray, 2013a; Binda, Pereverzeva, & Murray, 2014; Mathôt, van der Linden, Grainger, & Vitu, 2013; Naber, Alvarez, & Nakayama, 2013; Tkacz-Domb & Yeshurun, 2018; Unsworth, Robinson, & Miller, 2018), visual awareness in binocular rivalry conditions (Einhäuser et al., 2008; Fahle, Stemmler, & Spang, 2011; Kimura, Abe & Goryo, 2014; Naber, Frässle, & Einhäuser, 2011), perception of changes in stimuli low-level features, such as color or motion (Kohn & Clynes, 1969; Sahraie & Barbur, 1997; Ukai, 1985), perceptual illusions (Laeng & Endestad, 2012; Suzuki, Minami, Laeng, & Nakauchi, 2019; Zavagno, Tommasi, & Laeng, 2017), visual imagery (Laeng & Sulutvedt, 2014; Mathôt, Grainger, & Strijkers, 2017), and high-level processing of image content (Binda & Murray, 2015b; Binda, Pereverzeva, & Murray, 2013b; Naber & Nakayama, 2013; Sperandio, Bond & Binda, 2018). 

Particularly relevant to the present study are findings showing that the pupil does not constrict only in response to the physical luminance of a stimulus, but also in response to its perceived luminance. For example, Lang and Endestad (2012) found that optical illusions that induce a subjective impression of brightness (Kitaoka lightness illusion) elicit pupillary constriction, compared with control stimuli (Kanizsa form illusion), despite the actual luminance was controlled. Later, Laeng and Sulutvedt (2014) continued the research toward an increasingly abstract level of stimuli, showing that mentally visualizing a bright scene, compared with a darker scene, produces pupillary constriction. Recently, Suzuki et al. (2019) found that colorful glare illusions (especially blue), that subjectively enhance the perception of brightness, induce pupillary constriction, reflecting an adaptive response of the visual system to a probable dangerous situation of dazzling sunlight. Furthermore, Binda et al. (2013b) found that pictures of the sun induce pupillary constriction compared with control stimuli of matched luminance, as photographs of the moon, showing that high-level interpretations of image content can modulate the pupil response. Naber and Nakayama (2013) also investigated the pupillary responses to a variety of natural scenes with the same low-level features, demonstrating a larger amplitude of pupil constriction to scenes containing a sun. By showing inverted images, they also investigated the effect of contextual information on the pupil, demonstrating how visual complexity affects pupil size. Taken together, these findings confirm that pupillary responses to ambient light reflect the interpretation of the light in the scene and not simply the amount of physical light energy entering the eye. 

All of these studies indicate that the pupil diameter is sensitive to top-down modulation, and consequently that the pupil diameter could be modulated by cortical pathways other than the subcortical PLR system (Becket Ebitz & Moore, 2019; Binda & Murray, 2015a). A recent experiment (Sperandio et al., 2018) demonstrated that these extra-retinal modulations require visual awareness to modulate the pupil size. Using the continuous flash suppression (CFS) technique, they found that when participants were aware of sun pictures their pupils constricted relative to the control stimuli. This did not happen when the pictures were successfully suppressed from awareness, demonstrating that pupil size is sensitive to the contents of consciousness. 

In the present study, we measured the pupil response to artistic paintings representing scenes with a visible sun, a visible moon, or the presence of diffused light to address the effect of cognitive interpretation of very complex stimuli. In fact, paintings render a scene through the artist's mind, requiring an even higher level of interpretation compared with photographs or artificial stimuli (Altschul, Jensen, & Terrace, 2017; Tatler & Melcher, 2007). In addition to the effect of image content, we also investigated the effect of contextual information, such as color and global layout. We aim to confirm that the pupil size depends on complex features of the visual stimulus that are presumably processed in the cortical areas. 

The present study comprises one main and two control experiments to investigate effects of paintings categories, contextual information, and subjective interpretation (Figure 1). 

Effects of paintings’ categories

Three categories of paintings were used. Paintings of the sun were used to investigate if pictorial representations of high-luminance objects may elicit a smaller pupil size than other subjects, independently on the luminance of the images. Paintings of the moon were used to investigate pupillary response to stimuli representing a luminous disc as well but cognitively associated with a dark scene. Paintings with diffused light or different light sources (e.g. fires, volcanoes, etc.) were used to investigate if the mere presence of light in the absence of a luminous disc has any effect on pupil diameter. 

To ensure that the results have general meaning, for each category we have purposely chosen stimuli painted over a period of more than 300 years and pertaining to very different styles, and we think this represents a strong point of the study. 

In the main experiment (experiment 1), all the stimuli were presented by making them appear over a background of higher luminance. If the response depended only on overall light level, the same pupillary dilation would be expected for all stimuli. On the other hand, presentation of images depicting luminous objects is expected to produce pupil constriction due to high level visual processing (Binda et al., 2013b; Sperandio et al., 2018), over-riding the effect due to the physical properties of the stimulus. We expect to find a smaller pupil size for stimuli containing a light source, particularly the sun, due to the high-level interpretation of paintings content. 

In a second control experiment (experiment 2), to rule out possible effects of luminance on the results of the main experiment, stimuli were presented by making them appear over a grey background of matching luminance. In this condition, there is no discrepancy between the luminance of the screen during fixation and the stimulus, therefore, any deviation from baseline pupil size would be due to stimulus content only. As in the main experiment, we expect a smaller pupil size for stimuli with luminous light sources. 

Because studies have shown that pupillary responses are more sensitive to luminance changes in the fovea (Clarke, Zhang, & Gamlin, 2003), a third control experiment (experiment 3) was done by repeating the same paradigm of the main experiment, except stimuli were presented in the periphery of the visual field. We expect to confirm the results of the main experiment, ruling thus out as a possible dependence of pupillary response on retinal eccentricity. 

Effects of contextual information

Color and spatial layout of images are crucial tools for artists to enhance the aesthetic experience in paintings (Chatterjee & Vartanian, 2014; Graham & Field, 2008; Montagner, Linhares, Vilarigues, & Nascimento, 2016; Nascimento et al., 2017). Color is a very important feature for interpreting visual scenes (Goffaux et al., 2004, Greene & Oliva 2009; Oliva & Schyns, 2000; Oliva & Torralba, 2006; Steeves et al., 2004); however, the effect of variations in stimulus color on pupil responses has been suggested, but not systematically investigated (Snowden et al., 2016). Contextual cues, such as relative position of objects and their orientation, are undoubtedly important for fast image interpretation (Oliva & Torralba, 2006). Disrupting these cues can have an effect in pupillary response to images, as already shown by Naber and Nakayama (2013) with computer rendering of natural images. These variables were investigated within experiment 1 and experiment 2, by comparing pupil responses to original paintings (up-right and full-color) with their inverted (180 degree rotated) and no-color (grey-scale) versions. 

Effects of subjective interpretation

It is well known that aesthetic experience is unique to each individual (Kuchinke, Trapp, Jacobs, & Leder, 2009; Marković, 2010; Marković, 2011; Marković, 2012; Marković & Radonjić, 2008). It has also been shown that individual mental imagery (Laeng & Sulutvedt, 2014; Mathôt et al., 2017) and the content of consciousness (Sperandio et al., 2018) affect pupillary reactions. This means that the content represented in our paintings may be differently interpreted by each participant and, as a consequence, affect pupil diameter. For these reasons, we tested whether the paintings chosen as our stimuli elicited different pupil responses in experiment 1 depending on how the observer interpreted the scene, based on their response to a categorization test. 

Methods

Participants

Twenty-eight observers (18 women and 10 men, mean age = 27.2, SD = 5) participated in experiment 1, another 12 observers (5 women and 7 men, mean age = 26.4, SD = 4) participated in experiment 2, and another 12 observers (6 women and 6 men, mean age = 26.5, SD = 4) participated in experiment 3. Before starting the experiments, all participants filled out a questionnaire about personal data, presence of aberration or optical defects, history of brain damage, medication intake, tobacco consumption, and caffeine intake. All selected participants had a normal or corrected-to-normal vision (by contact lenses) and did not take any type of medication. Participants were asked to abstain from drinking coffee before the experiment and not to wear eye make-up. Observers were unaware of the aim of the experiment and gave written informed consent before the experiment. All experimental procedures were approved by the local ethics committee (Comitato Etico Pediatrico Regionale – Azienda Ospedaliero-Universitaria Meyer – Firenze FI) and were compliant with the Declaration of Helsinki. 

Apparatus and set-up

Each participant was tested individually in a dark room, with no lighting other than the display screen. Stimuli were presented on an ASUS monitor (51 × 29 cm, resolution 1920 × 1080 pixels), through a dedicated computer (iMac Retina 5K, 27-inch, mid 2015 3.3 GHz Intel Core i5 processor, MacOs Sierra software version 10.12.6). The observer was positioned at 57 cm distance from the monitor with a chin rest used to stabilize the head. Pupil diameter was binocularly tracked at 60 Hz with a CRS LiveTrack FM system (Cambridge Research Systems). Stimulus presentation and data collection programs were developed using Matlab (R2016b version). 

Stimuli

We selected 30 paintings of natural scenes, produced in different historical periods (1700–2000) and with different styles (impressionism, realism, etc.). Each stimulus was nominally assigned to one of the three categories of our study, based on circumstantial elements, such as painting's title or the authors’ interpretation (Table 1; for examples of each category see Figure 2A). All images were resized (conserving proportions) to either a width or a height of 283 pixels, with the other side ranging from 178 to 355 pixels. The original luminances of all paintings, in all their versions, were modified and were rescaled to the same value, corresponding to the average luminance of the whole set (9.7 cd/m²). They were also rescaled to a common resolution (28.35 pixels/cm). The luminance varied within each image, reaching its maximum at the point where the source of illumination was represented. We measured the value of luminance at the center of each lunar/solar disc represented in our images, and tested for differences between sun and moon distributions, finding no statistically significant effect (sun: M = 40.2 cd/m², SD = 13.7 cd/m²; moon: M = 37.5 cd/m², SD = 17.3 cd/m²; t(1) = 0.38, p > 0.05; Figure 3). 

Table 1.

 

List of paintings used as stimuli.

 

In addition to the 30 paintings, a set of 10 uniform-grey rectangular images were generated, matching the mean luminance (9.7 cd/m²) and the average size of paintings, to be used as control stimuli for luminance. 

Furthermore, a grey-scale and an inverted (180 degree rotated) version were produced for each painting (see Figure 4A). They were used in experiments 1 and 2, to assess the role of color and global image organization. 

Procedure

The eye tracker was calibrated at the beginning of each session with a standard 9-point calibration routine. In experiment 1, trials started with the presentation of a black fixation cross (5 × 5 mm) in the center of a white screen (71 cd/m²) for 2.5 seconds (pre-stimulus interval). This was followed by the presentation of one of the stimuli for 2 seconds (stimulus interval). The fixation cross was kept visible in the center of the screen during the pre-stimulus and stimulus intervals, whereas the luminance of the background screen was kept constant at 71 cd/m². Observers were instructed to keep their gaze at the fixation cross for the whole of the pre-stimulus and stimulus intervals, refraining from blinking, and not to perform any other task. During this time, pupil size was continuously monitored by means of a camera attended by the experimenter on her own screen (using QuickTime software) throughout the whole experiment. Each trial was followed by an inter-trial interval of 2 seconds, in which a white screen (71 cd/m²) was displayed. During this time, the eye tracker did not record, and the observers were allowed to blink and rest their eyes before the next trial (Figure 1A). 

Figure 1.

 

Schematic of experimental procedures. Procedure used in experiment 1 (A), experiment 2 (B), and experiment 3 (C). Copyright permission from the Author was obtained for the painting shown, Rural sunrise (Gercken, 2012).

Experiment 1 consisted of 100 trials divided into 4 blocks of 25 images: 10 different paintings per category plus their inverted and grey-scale versions, plus 10 uniform-grey control stimuli. The sequence of stimuli presentation was randomly predetermined and kept the same for all observers. 

Experiment 2 followed the same procedure of experiment 1, except that stimuli were presented on a grey background having the same luminance as the mean luminance of the stimuli (9.7 cd/m²; Figure 1B). In this experiment, uniform grey control stimuli, having the same luminance as the background, were not used. This led to 90 trials in 2 blocks of 22 plus 2 blocks of 23 stimuli. 

Experiment 3 also followed the same procedure of experiment 1, but stimuli were presented in an off-center location, 5 degrees to the right of the fixation cross (Figure 1C). In this case, grey-scale and inverted versions of paintings were not tested, leading to 40 trials divided in 2 blocks of 20 stimuli. 

After the experiments, all paintings were presented again in sequence to the observers without time limitation and pupil recording, asking them to categorize each, as “sun,” “moon,” or “other.” The complete procedure took about 50 minutes per observer, of which about 30 minutes were of pupil recordings. 

Data processing

Raw data recorded by the eye-tracker were processed in the same way for all three experiments. Right and left pupil diameters were averaged, and the resulting value was transformed from pixels to millimeters. Calibration was attained by measuring the instrument's recording of a 4 mm artificial pupil, positioned at the approximate location of the subjects’ left eye. 

For each observer, a baseline pupil diameter was calculated by averaging pupil diameter recorded over the last 500 ms of the pre-stimulus interval in each trial. This baseline value was then subtracted from each recording of that observer over the whole 4.5 second period (Mathôt, Fabius, Van Heusden, & Van der Stigchel, 2018). 

All results were classified according to the categorization made by the observer in the test, to ensure that the pupil size corresponded to the subjective interpretation of the nature of light source. For example, if a painting with a moonlit scene had been categorized as “sun” by some participants, the recordings obtained with this image were analyzed as a sun stimulus for this observer. 

The analysis of the pupil responses elicited by different categories of paintings, or different versions of the same painting, follows a method widely used in literature for this type of experiments (Binda et al., 2013b; Naber & Nakayama, 2013). An average pupil size µ was calculated for each image category as follows. First, all recordings from each observer s p_s,i(t), where i is the stimulus index, were averaged as a function of time \(\begin{array}{@{}l@{}} \overline {{p_s}} (t) = \frac{{\sum\nolimits_i^I {{p_{s,j}}(t)} }}{I}\\ \end{array}\) (I = 10 for each category). Then, temporal averages were computed over the duration of the stimulus interval for each observer \({\mu _s} = \frac{{\sum\nolimits_{t = 0}^T {\overline {{p_s}} (t)} }}{T}\), from which the overall average was computed for each category as: \({\mu _{}} = \frac{{\sum\nolimits_{s = 1}^N {{\mu _s}} }}{N}\). This quantity was attributed an overall variance SE \(\sigma _{}^2 = \frac{{\sum\nolimits_{s = 1}^N {{{({\mu _s} - \mu )}^2}} }}{{N - 1}}\) . Differences between µ of different categories were assessed with ANOVA and pairwise comparisons were done with post hoc Student's t-tests with Bonferroni corrections. 

Moreover, the response functions for each category \({\overline p _s}(t)\)were averaged over the N observers, to obtain time-dependent averages \(\overline p (t) = \frac{{\sum\nolimits_s^N {{{\overline p }_s}(t)} }}{N}\) together with their time-dependent standard errors \(SE(t) = \sqrt {\frac{{\sigma _{}^2(t)}}{N}} = \sqrt {\frac{{\sum\nolimits_{s = 1}^N {{{(\overline {{p_s}} (t) - \overline p (t))}^2}} }}{{N(N - 1)}}} \). 

In addition, data were also analyzed on an image by image basis as follows. For each image i, the response for each participant s as a function of time, p_s,i(t), was averaged over the stimulus interval to yield \({\mu _{s,i}} = \frac{{\sum\nolimits_{t = 0}^T {{p_{s,i}}(t)} }}{T}\) and then over all participants to yield the time-average response for each image \({\mu _i}\frac{{\sum\nolimits_{s = 1}^N {{\mu _{s,i}}} }}{N}\), with an associated standard error \(SE_i^{} = \sqrt {\frac{{\sum\nolimits_{s = 1}^N {{{({\mu _{s,i}} - {\mu _i})}^2}} }}{{N\sqrt {N - 1} }}} \). 

Results

Effects of paintings’ categories

The main result of this work comes from the comparison of responses to the presentation of the three categories of paintings and to the uniform grey control stimuli. The time course of pupil size for each painting category \(\bar p(t)\) obtained from experiment 1 is shown in Figure 2B (left). Because all images equally and greatly reduce the luminance level across the screen, if the response were based only on luminance, we would expect the same pupillary dilation for all categories. In fact, the line graph in Figure 2B (left) shows that sun stimuli elicited a much smaller dilation than all other categories, despite having the same mean luminance. Paintings with the moon, paintings with diffused light, and uniform grey control stimuli induced a consistent pupillary dilation. 

Figure 2.

 

Mean pupillary responses to different paintings categories. (A) Examples of paintings belonging to the four categories of stimuli. Example of sun painting: Aurora (Bierstadt, 1850); example of moon painting: Astronomical Observations: the Moon (Creti, 1711); example of diffused light painting: Landscape at Giverny (Monet, 1888). Paintings shown are in the public domain. (B) Experiment 1. Left: Baseline-corrected pupil size \(\overline p (t)\), for the four stimulus categories, plotted as a function of time from trial onset. Right: µ of different categories. (C) Experiment 2. Left: \(\overline p (t)\)for the three stimulus categories. Right: µ of different categories. (D) Experiment 3. Left: \(\bar p(t)\) for the four stimulus categories. Right: µ of different categories. The vertical line in the graphs on the left indicates stimulus onset. Error bars on the left are SE(t). Error bars on the right are SE of the means µ. Red: sun; blue: moon; green: diffused light; black: grey-uniform control stimuli. Asterisks mark statistically significant pairwise comparisons across image categories: *p < 0.05; **p < 0.01, ***p < 0.001. All data shown have been corrected based on each observer's categorization.

Significant differences between all categories of stimuli µ are evidenced by ANOVA (F(3) = 20.54, p < 0.001). Pairwise comparisons (Table 2) show that paintings with the sun produced lower dilation than paintings with the moon, with diffused light and uniform luminance images. In addition, moon paintings produce smaller dilation than uniform grey control stimuli. No statistical difference is found between the dilation induced by diffused light paintings and moon or uniform grey control stimuli (see Figure 2B, right). 

Table 2.

 

Statistics tests for effects of paintings categories.

 

The size of differences between conditions, estimated by Cohen's d statistics, is very small for sun versus moon paintings (s = 0.55, d = 0.12), small for sun versus diffused light paintings (s = 0.55, d = 0.18), and sun versus uniform-grey (s = 0.53, d = 0.22), and very small for moon versus uniform grey (s = 0.53, d = 0.09). Values lower than 0.01 were considered to be negligible effects (Cohen, 1988; Savilowsky, 2009). 

The time course \(\overline p (t)\) also show the same general trend for all categories (see Figure 2B, left). Pupil diameter increases gradually during the pre-stimulus interval, then remains stable for about 500 ms after stimulus onset, at a common level for all categories. After this, pupil size starts to increase with different slopes according to different stimulus categories. The associated uncertainty SE(t) also increases with time for painting stimuli, while staying approximately constant for control stimuli (see the Discussion section for possible explanations). This highlights the advantage pertaining to the second method of analysis, whereby different data points are combined with proper accounting for their differing uncertainties. 

Because eye movements can influence pupil changes (Gagl, Hawelka, & Hutzler, 2011), although observers were instructed to keep fixation and their eye movements were monitored, we analyzed a posteriori the average position of their eyes with respect to the fixation cross for the different stimulus categories. The average distance from fixation in millimeters was minimal (sun: 2.45 ± 0.5; moon: 2.88 ± 0.6; diffused light: 2.09 ± 0.4; and mean luminance: 2.59 ± 0.5) and the same for all categories, included the uniform grey stimuli (ANOVA, F(3) = 0.38, p > 0.05). 

Results of experiment 2 are displayed in Figure 2C. In this case, the same pupillary constriction is expected for all kinds of paintings, but we found that the constriction induced by paintings of the sun is larger than those elicited by paintings of the moon and paintings with diffused light (ANOVA (F(2) = 11.88, p < 0.001; see Table 2). The size of this effect is categorized as small for sun versus moon paintings (s = 0.71, d = 0.2) and sun versus diffused light (s = 0.69, d = 0.3). 

In experiment 3, where paintings are displayed in the periphery of the visual field, the time course of responses (Figure 2D, left) suggests a lower dilation for paintings of the sun than for other categories. This is confirmed by the ANOVA analysis (F(3) =9.86, p < 0.001; see Table 2; Figure 2D, right). The size of this effect is very small for sun versus moon paintings (s = 0.51, d = 0.1), sun versus grey uniform (s = 0.49, d = 0.1), and small for sun versus diffused light (s = 0.54, d = 0.2). 

Image by image analysis

Paintings are less uniform stimuli than photographs in representing a given subject. To assess the variance of the responses elicited by different paintings, data of experiment 1 have been analyzed image by image, and results are shown in Figure 3A. 

Figure 3.

 

Pupillary responses to individual stimuli. (A) Round filled symbols are average responses for each image, µ_i, of observers that classified the paintings according to the nominal classification given by the authors. Red: sun paintings; blue: moon paintings; green: diffused light paintings and grey: uniform-grey control stimuli. Hollow squares are individual responses of observers that did not classify the paintings according to the nominal classification. Red: painting classified as sun, green: painting classified as other. Error bars are \(SE_i^{}\). Locations of misinterpretations in the distribution of each image are Image 3: 82^nd percentile; image 6: 96th percentile; image 8: 57th < percentile < 93rd; image 17: 11th < percentile < 39th; image 18:11th < percentile < 46th; image 19: 21st percentile; image 23: 11th < percentile < 39rd; image 25: 18th percentile; image 27: 39th percentile; image 28: 21st percentile; image 30: 32nd percentile. (B) Correlation between the local luminance at the center of the light source of each painting and the corresponding pupillary response averaged across observers µ_i (experiment 1). There is no significant correlation between pupil dilation and local luminance at the center of suns (R² = 0.23, F (1) = 3.83, p > 0.05) or moons (R² = 0.06, F (1) = 0.45, p > 0.05). The dotted lines indicate the mean luminance in the center of sun (red, M = 40.2 cd/m², SD = 13.7 cd/m²) and moon paintings (blue, M = 37.5 cd/m², SD = 17.3 cd/m²). All data shown have been corrected based on each observer's categorization.

The first finding is that the large majority of images were classified by observers in agreement with the nominal classification provided by the authors, but there were a small number of exceptions. They occur in 11 paintings, for a total of 20 observations, amounting to 2% of total occurrences. They are an interesting effect that we investigate further in the section Effects of subjective interpretation below, but their limited number has a small effect on the overall results, as we verified by repeating the analysis based on the nominal rather than the observers’ classification. 

For the cases where the paintings were perceived according to their nominal categorization, the variances in pupil responses (\(\sigma _{SUN}^2 = 0.002\), \(\sigma _{MOON}^2 = 0.003\), \(\sigma _{DIFFUSED}^2 = 0.002\), \(\sigma _{GREY}^2 = 0.001\)) are compatible among all stimulus categories (Fisher's tests, p > 0.1 for all comparisons). More importantly, they are also statistically compatible with the variance of the responses observed to uniform grey control stimuli (Fisher's tests, p > 0.1 for all comparisons). This indicates that the obvious differences between individual paintings do not dominate the observed spread in response. 

For the cases where the paintings were not perceived according to their nominal categorization, pupil responses were always in the direction of the average of the perceived stimulus: when sun paintings, were perceived as other, pupil sizes were larger, when moon and diffused light paintings were perceived as sun, pupil sizes were smaller. However, values, although apparently off-scale, were all comprised within 11th and the 93rd percentiles of image distributions (for all values, see the caption of Figure 3A). 

All stimuli had the same mean luminance, but they depict light sources of different size and intensity. To control for dependence on these variables, measurements in experiment 1 were correlated with the luminance value in the center of the light source. Figure 3B shows no significant correlation between pupil dilation and local luminance at the center of suns (R² = 0.23, F (1) = 3.83, p > 0.05) or moons (R² = 0.06, F (1) = 0.45, p > 0.05). In addition, no statistical difference is seen between average local luminance values at the centers of the sun and moon light sources (t(1) = 0.38, p > 0.05). 

Effects of contextual information

Another interesting result of experiment 1 follows from the comparison between pupillary response elicited by paintings of the sun in their original, grey-scale, and inverted versions (examples in Figure 4A). The graph Figure 4B (left) shows the time course of pupil size \(\overline p (t)\) for sun paintings, and their grey-scale and inverted versions. Average pupil responses µ are found to be different between these three conditions (ANOVA: F(2) = 28.09, p < 0.001). Grey-scale and inverted versions produce a significantly wider pupillary dilation than the original version of the sun paintings. This suggests that manipulations of image structure or color may alter the interpretation of scene brightness and, as a consequence, modulate the pupil response itself. In addition, grey-scale versions produce a larger dilation than inverted versions of the paintings. This indicates that the global arrangement of painted elements is less important than their color in suggesting the presence of light in a painting (Table 3; Figure 4B, right). The size of these differences, assessed by Cohen's d, is very small for original versus inverted versions (s = 0.54, d = 0.13) and inverted versus grey-scale (s = 0.54, d = 0.08), and small for original versus grey-scale versions (s = 0.55, d = 0.21). ANOVA shows statistical differences also for different versions of diffused light paintings (ANOVA: F(2) = 5.10, p < 0.01). Indeed, grey-scale versions of diffused light paintings produce more dilation than their original versions (t(2) = 3.04, p < 0.05). Instead, responses to different versions of moon paintings are not statistically different (ANOVA: F(2) = 1.87, p > 0.05). 

Figure 4.

 

Mean pupillary responses to different versions of paintings of the sun. (A) Example of a sun painting in original, inverted and grey-scale version. Copyright permission from the author was obtained for the painting shown, Rural sunrise (Gercken, 2012). (B) Experiment 1. Left: Baseline-corrected pupil size \(\overline p (t)\), for the three versions of sun paintings, plotted as a function of time from trial onset. Right: µ of different versions. (C) Experiment 2. Left: \(\overline p (t)\) for the three versions of sun paintings. Right: µ of different versions. The vertical line in the graphs on the left indicates stimulus onset. Error bars on the left are the SE(t). Error bars on the right are the SE of the means µ. Red: original versions of sun paintings; red/white: inverted versions of sun paintings; grey: grey-scale versions of sun paintings. Asterisks mark statistically significant pairwise comparisons across image categories: *p < 0.05; **p < 0.01, ***p < 0.001. All data shown have been corrected based on each observer's categorization.

Table 3.

 

Statistics tests for effects of contextual information.

 

Although the same observer sees the same painting only once in the original, once in the reversed, and once in the grey-scale version, that are different for contextual information, there still may be a habituation effect on pupil size as described by Yoshimoto, Imai, Kashino, and Takeuchi (2014). A 2-way ANOVA ruled out this possibility showing a significant main effect of sun paintings’ versions (ANOVA: F(2) = 28, p < 0.001) but no significant effect of order presentation (F(2) = 1.28, p > 0.05). 

The same pattern of results is obtained with the same stimuli in experiment 2 (see Figure 4C, left). Original versions of sun paintings elicit more constriction than their inverted versions, that in turn elicit more constriction than grey scale versions (ANOVA: F(2) = 33.14, p < 0.001; see Table 3; Figure 4C, right). The size of these differences, assessed by Cohen's d, is small for original versus inverted versions (s = 0.70, d = 0.2) and original versus grey-scale (s = 0.70, d = 0.3), and very small for inverted versus grey-scale versions (s = 0.70, d = 0.11). ANOVA shows statistical differences also for different versions of moon (ANOVA: F(2) = 5.96, p < 0.01) and diffused light paintings (ANOVA: F(2) = 15.48, p < 0.001) . Indeed, grey-scale versions of moon paintings produce less constriction than their original versions (t(2) = 2.96, p < 0.05), and grey-scale versions of diffused light paintings produce less constriction than their original (t(2) = 5.11, p < 0.001) and inverted versions (t(2) = 4.57, p < 0.01). Therefore, in this condition, for all stimulus categories, the disruption of contextual cues alters pupillary response. 

Effects of subjective interpretation

Paintings are intrinsically complex stimuli, requiring a greater interpretative effort when compared with photographs and real-life scenes, leading to cases of ambiguous interpretation by observers. This is the reason for performing our main analysis based on individual observers’ response to the categorization test (see Procedure). It is, however, interesting to look in more detail to the cases of ambiguous response. Figure 3A shows, image by image, not only the average response of conformant observations, but also displays the individual responses observed in the few cases of nonconforming categorizations. Inspection of Figure 3A clearly suggests that when observers classified a nominal sun painting as “other” (therefore, they did not see any light source) their pupil got a larger pupil size than that of those that had classified the same image as sun; whereas moon and diffused light paintings elicited a smaller pupil size in observers that had classified them as “sun” stimuli. 

To test for the presence of the effect of subjective image interpretation, a nonparametric, one-tailed, Mann-Whitney ranking test was performed for data of all paintings that elicited differing responses in our experiment (in cases where only one misinterpretation occurred, the p value was directly determined as the ratio of the rank of the outlier and total number of subjects). Results show a significant effect for each case tested (p < 0.05). To assess the overall significance for the presence of an effect, individual p values were combined according to the Fisher's method (Mosteller & Fisher, 1948) yielding an overall p value < 0.0001. This is a strong indication for an influence of cognitive interpretation of a visual scene on the pupillary response of the observer. 

Figure 5 shows, as an example, µ_i for the three most ambiguous stimuli of our set, each receiving 3 of 4 misclassifications in experiment 1, reported according to the categorization received (“sun,” “moon,” or “other”). 

Figure 5.

 

Effects of subjective interpretation. Single observer and average pupillary response µ_i (mm) for three stimuli subjected to three or more misinterpretations. Classification is based on the categorization of the light source made by the participants in the test. Blue: categorization as a moon, red: categorization as a sun, green: categorization as other. (A) Moon (n = 25): 0.06 ± 0.3; sun (n = 3): −0.09 ± 0.06. (B) Moon (n = 24): 0.8 ± 0.04); sun (n = 4): −0.0.8 ± 0.04. (C) Other (n = 24): 0.14 ± 0.03); sun (n = 4): 0.01 ± 0.03. Error bars are the SE_i. Asterisks mark statistically significant comparisons between groups, non-parametric one-tailed Mann-Whitney test, p < 0.05*. Painting in A (image 17: The Starry Night, van Gogh, 1889) is in the public domain; copyright permission from the Author was obtained for painting in B (image 18: Harvest moon, Solberg, 2012); painting in C (image 23: The slave ship , Turner, 1840) is in the public domain.

Discussion

We show that artistic paintings, depicting scenes illuminated by light sources of different nature, such as the sun, moon, or containing a diffused lighting, although much less realistic than photographs in representing natural scenes and largely mediated by the artist’s interpretation of reality and his technique, can differently modulate the pupillary response, according to the scene represented and not to their specific luminance or other low-level visual features. 

In fact, despite that all paintings had the same mean luminance, when presented on a lighter background, paintings containing a light source produced less dilation than meaningless mean grey uniform-luminance rectangles, representing the control for dilation in this condition. In particular, paintings with the sun elicited a much smaller dilation than paintings with the moon, that in turn produced a lower dilation than paintings containing no visible light source. 

This pattern of results does not depend on background luminance. When paintings are presented on a mean grey background, all produce constriction, although not expected from their average luminance that is equivalent to the background. This is in agreement with previous observations of the onset of changes in contrast, besides luminance, eliciting pupillary constriction (Naber et al., 2011, Naber & Nakayama, 2013). We find that the constriction induced by paintings containing a visible sun is larger than that produced by moon and diffused light paintings. 

It is well known that the strength of pupillary response is larger for luminance changes occurring in the fovea (Clarke, Zhang, & Gamlin, 2003), and this raises the question of the role played by the higher values of luminance found in the vicinity of the fixation center in the case of sun and moon paintings. The fact that spatial distribution of luminance in the visual field and between image categories is not responsible for the observed differences between categories is demonstrated by three independent observations. First, when paintings are presented in the periphery, the same patterns of results are obtained: sun paintings produce less dilation than moon, diffused light and grey uniform control stimuli. This is in agreement with previous findings on photographic images (Binda et al., 2013b). Second, no correlation was found between pupil dilation and the local luminance measured at the center of suns or moons. Finally, the average luminance at the centers of sun and moon disks are compatible. 

All the effects found for different stimulus categories do not depend on eye movements that have been shown to modulate pupil response (Gagl et al., 2011). 

Our findings are in general agreement with those reported in the literature with non-painting stimuli (Binda et al, 2013b; Naber & Nakayama, 2013), but sun paintings produce a weaker effect compared with realistic pictures (Binda et al., 2013b). This might be the result of several factors, like differences in stimulus size and relative difference between luminance of stimuli and background. Our stimuli are also much more complex and may require higher cognitive load (Altschul et al., 2017; Tatler & Melcher, 2007), which is known to cause pupil dilation (Beatty, 1982; Hess & Polt, 1964; Just & Carpenter, 1993). 

Results do not depend on the specific paintings chosen for the experiments, assigned to the three categories by the experimenters, and validated by all subjects in the categorization test. Although photograph categories chosen in similar studies comprise more or less homogeneous sets (see Binda et al., 2013b), here, paintings in the same category have been deliberately chosen to be as different as possible in style and period, to ensure the general validity of the findings. Despite this diversity, variability of responses to sun, moon, and diffused light paintings are the same, and, more importantly, they do not differ from the variability of responses to uniform grey control stimuli. This indicates that the pupil response is mainly driven by the scene depicted, overstepping differences in painting styles, artist's personal style, or his/her technique rendering of light sources. 

Interesting results emerge also from the analysis of time variation of pupil size in experiments with light background. During the pre-stimulus interval there is a gradual increase of pupil diameter, possibly due to the effect of expectations (Irons, Jeon & Leber, 2017). During the first 500 ms after stimulus presentation, pupil diameter is mostly stable and equal for all the categories. This could be because the constriction that usually occurs when a stimulus appears (Naber & Nakayama, 2013; Naber et al., 2011; Privitera, Renninger, Carney, Klein, Aguilar, 2010) may be compensated by the dilation that should be produced by showing a stimulus darker than background. After this 500 ms period, pupil response starts to differ between categories. For all of them, though, there is a progressive increase of pupil size up until the end of the recording, consistent with the dilation effect due to cognitive load described in literature (Hess & Polt, 1964; Just & Carpenter, 1993; Kahneman & Beatty, 1966). 

Interestingly, the variability of observers’ responses to all categories of paintings also increases with time, being larger for sun paintings, whereas remaining more or less constant for the response to the uniform-grey control stimuli. Note that this same effect was also present in pupil responses to photographs (Binda et al., 2013b) or to words conveying a sense of brightness or darkness (Mathôt et al., 2017), although not analyzed or commented by the authors. We cannot be sure about the cause of this effect, but we could speculate that a number of different cognitive processes progressively set in while observers keep looking at the stimuli. This may include attention, recognition of elements in the painting, familiarity with the specific painting, aesthetic preference, memory, imagination, etc. All these factors, being different for each individual, produce a larger variability of responses than the one that could be generated by lower level perceptual visual mechanisms. This hypothesis is also in agreement with the observation that uniform-grey images, not involving such high-level processes, do not exhibit the same increase in variability. 

Inverted paintings of the sun produce a larger pupil size than originals, despite sharing the same low-level features, such as luminance, contrast, chromatic contrast, and Fourier transform. This shows again that pupil amplitude is largely modulated by the observer's interpretation of the luminous objects rather than by its low-level features (Binda et al., 2013b; Naber & Nakayama, 2013). Image inversion is known to impair recognition performance of stimuli, such as pictures of faces, buildings, and cartoons (Naber & Nakayama, 2013; Scapinello & Yarmey, 1970; Strother et al., 2011; Valentine & Bruce, 1986; Van Belle, De Graef, Verfaillie, Rossion, & Lefevre, 2010; Yin, 1969). Therefore, by changing the complex relations between features of the paintings, the information about its content decreases, making it more difficult for the observer to use contextual cues to identify the source of light. A similar effect was found by Naber and Nakayama in computer generated images (Naber & Nakayama, 2013). 

Grey-scale versions of sun paintings cause an even greater pupil size than originals, comparable to that produced by uniform-grey images, devoid of meaning, used as controls. Because chromatic content is a very important cue used for image interpretation (Goffaux et al., 2004, Greene & Oliva 2009; Oliva & Schyns, 2000; Oliva & Torralba, 2006; Steeves et al 2004), the fact that the absence of color in sun paintings increases pupil size is further proof of pupillary response being largely driven by interpretation of the light source. The suggestion that colored stimuli may produce different pupil response than their grey-scale versions was indeed previously advanced, although not systematically investigated (Snowden et al., 2016). 

The grey-scale versions of our sun stimuli also cause a larger pupil size than inverted versions, suggesting that color cues are even more important than spatial organization for the identification of the light source. 

Note that the presentation of each painting in three different versions does not affect pupil responses, as expected with multiple exposures to the same stimulus (Yoshimoto et al., 2014), probably because the three versions are not perceived as repetitions of the same stimulus. 

The chromatic structure of artistic compositions mostly follows the statistical features of the natural environment (Montagner et al., 2016). Therefore, blue colors are generally used in night scenes representations, whereas yellow-reddish chromaticities are used in rendering daylight scenes. Thus, different response to moon and sun paintings might be ascribed to their different chromatic contents. However, the results of this work imply that the presence of an object interpretable as a light source plays a crucial role in scene reconstruction. Indeed, diffused-light paintings endowed with the same yellow-reddish chromaticities of sun paintings, but no visible light source, produce distinguishably larger pupil size. 

Perhaps the most convincing evidence presented in this work for the crucial role of image interpretation in pupillary response, is the strong relationship observed between pupil diameter of observers and their subjective interpretation of the light source. The same painting is capable of eliciting constriction in observers who see it as a sun representation and dilation in those who see it as a moon. 

Other authors have tried to explain why showing images with a sun produces more constriction than images of the same luminance with different lighting structure, and we can reasonably presume that these explanations may hold also for the effects found with our paintings. A potential explanation is that the subjective perception of increased brightness reduces pupil size, as found with illusions by Laeng and Endestad (2012) and Suzuki et al., (2019) with psychophysical methods. Nevertheless, Binda et al., (2013b), by using a rating method of their stimuli, did not find this correlation. Moreover, Naber and Nakayama (2013) demonstrated that even cartoon depictions of the sun, appearing no brighter than cartoon depictions of the moon, can result in pupil constrictions. Another proposed explanation is based on different spatial distribution of attention across image categories, as it is known that attention strongly affects pupil size (Binda et al., 2013a). The observer's attention might be focusing more on the brighter regions of the sun pictures and spread more evenly in other images. However, this hypothesis has been ruled out by Binda et al., (2013b), showing that photographs of the sun cause constriction even when the observer's attention is directed to performing a different task. An explanation that still remains open after the present work is that of a protective behavior against a potentially harmful light level triggered by high-level interpretation of a very luminous object (Binda et al., 2013b; Laeng & Endestad, 2012; Naber & Nakayama, 2013; Suzuki et al., 2019). In other words, we can hypothesize that our system initiates a defense response to the powerful light induced by the sun, even if it is just depicted in a painting. 

All evidences presented in this work converge with the results of previous studies in suggesting a top-down control on the pupillary light reflex (Becket Ebitz & Moore, 2019; Binda & Murray, 2015a). The neural pathways underling this high-level modulation of PLR cannot be identified with certainty, but some potentially relevant circuits have already been identified. It is well established that pupillary constriction results from the activation of the subcortical Edinger-Westphal nucleus (EW) (Gamlin & Clarke, 1995), and there are some known modulatory inputs from cortical areas to this circuit. First, EW activity is enhanced by inputs from the visual cortex (Becket Ebitz & Moore, 2017; Binda & Gamlin, 2017) and the superior colliculus (Gamlin, 2006; Joshi & Gold, 2019; Joshi, Li, Kalwani, & Gold, 2016; Wang & Munoz, 2015; Wang & Munoz, 2012). Other possible inputs to the PLR could come directly from the prefrontal cortex, in particular from the frontal eye field (FEF), or indirectly through the extrastriate cortex, the oculomotor regions in the parietal cortex and the superior colliculus that are modulated by FEF (Becket Ebitz & Moore, 2017). EW nucleus also receives inhibitory input from the sympathetic system through projections from locus coeruleus (Joshi et al., 2016; Peinkhofer et al., 2019) and the hypothalamus that are potentially under cortical control (Aston-Jones & Cohen, 2005). A reduction of this inhibitory inputs could result in a pupillary constriction (Joshi & Gold, 2019; Wilhelm, 2002). 

Conclusions

The present work provides further evidence for the influence of high-level visual processing on the modulation of pupil response, corroborating an existing body of evidence. However, our specific choice of paintings as stimuli allows to push the exploration of the involved top-down mechanisms toward the even higher-level cognitive processing involved in aesthetic experience, imagination and memory. This reaches a point where the very same image can produce opposite responses depending on the individual subjective interpretation and visual awareness. 

Overall, this suggests that variations of pupil diameter can be an effective probe into cortical processing, making pupillometry a useful tool for the study of high-level vision and cognition. 

Acknowledgments

The authors thank Paola Binda e Marco Turi for helpful discussion and comments on the data. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (Grant Agreement No 832813 GenPercept “Spatio-temporal mechanisms of generative perception”). 

Commercial relationships: none. 

Corresponding author: Maria Michela Del Viva. 

Email: [e-mail]. 

Address: Department of Neurofarba, University of Florence, Florence, Italy.