Pupil size signals mental effort deployed during multiple object tracking and predicts brain activity in the dorsal attention network and the locus coeruleus

Abstract

Abstract

Abstract:

Abstract  Attentional effort relates to the allocation of limited-capacity attentional resources to meet current task demands and involves the activation of top-down attentional systems in the brain. Pupillometry is a sensitive measure of this intensity aspect of top-down attentional control. Studies relate pupillary changes in response to cognitive processing to activity in the locus coeruleus (LC), which is the main hub of the brain's noradrenergic system and it is thought to modulate the operations of the brain's attentional systems. In the present study, participants performed a visual divided attention task known as multiple object tracking (MOT) while their pupil sizes were recorded by use of an infrared eye tracker and then were tested again with the same paradigm while brain activity was recorded using fMRI. We hypothesized that the individual pupil dilations, as an index of individual differences in mental effort, as originally proposed by Kahneman (1973), would be a better predictor of LC activity than the number of tracked objects during MOT. The current results support our hypothesis, since we observed pupil-related activity in the LC. Moreover, the changes in the pupil correlated with activity in the superior colliculus and the right thalamus, as well as cortical activity in the dorsal attention network, which previous studies have shown to be strongly activated during visual tracking of multiple targets. Follow-up pupillometric analyses of the MOT task in the same individuals also revealed that individual differences to cognitive load can be remarkably stable over a lag of several years. To our knowledge this is the first study using pupil dilations as an index of attentional effort in the MOT task and also relating these to functional changes in the brain that directly implicate the LC-NE system in the allocation of processing resources.

Introduction

Attention can be thought of as the brain's allocation of processing resources to task relevant stimuli, which is controlled by modulatory changes in the brain's state of arousal (Coull, 1998). The allocation of processing resources or capacity is the intensive aspect of attention (Kahneman, 1973), which has been a less studied area compared to its selective aspects (Laeng, Sirois, & Gredebäck, 2012; Raizada & Poldrack, 2008). The related term “mental effort,” though difficult to define theoretically and experimentally (Sarter, Gehring, & Kozak, 2006), has been increasingly used in the psychological literature when referring to the intensive aspect of attention. Kahneman (1973, 2011) has metaphorically likened the allocation of attentional resources to an electricity grid: When a device (e.g., a toaster) is connected to the power grid, it expends resources from the grid to be able to do work. To compensate for the increased consumption, an increase in energy production takes place. Thus, the term “mental effort,” similarly to the concept of physical energy (cf. Friston, 2009, 2010), captures the intensive aspect of attention-demanding processes; as task demands increases—for example, by increasing the number of target objects to be simultaneously attended—so increases the required amount of attentional resources needed to perform the task successfully. 

According to the classic formulation by Kahneman (1973), arousal is a physiological dimension that can be measured, and mental effort is a special form of arousal that reflects both phenomenologically and psychophysically the variations in processing demands. In particular, Kahneman (1973) and Beatty (1982) proposed that the best available physiological index of changes in mental effort, either within and between cognitive tasks, as well as for between-subject differences, is given by observing task-evoked changes in pupil diameter. A large body of studies has confirmed in the last 50 years the usefulness of measuring pupillary changes to index mental effort, showing that the pupil dilates as a function of task difficulty across a wide range of task domains (Beatty & Kahneman, 1966; Bradshaw, 1968; Hess & Polt, 1964; Hyönä, Tommola, & Alaja, 1995; Just & Carpenter, 1993; Kahneman & Beatty, 1967; Karatekin, Couperus, & Marcus, 2004; Laeng et al., 2012; Stanners, Coulter, Sweet, & Murphy, 1979). Pupil size measures also reflect individual differences in the recruitment of attentional capacity, since smaller pupillary dilations (indicative of more spare capacity) are observed for more intelligent as compared to less intelligent participants (Ahern & Beatty, 1979, 1981), and also for younger versus older participants, at a given task difficulty (Piquado, Isaacowitz, & Wingfield, 2010). In other words, pupil size can be used as a signal of the relative amount of effort needed to accomplish a task. Thus, it has been suggested that pupil diameter might be used as a summative index of brain activity (Siegle, Steinhauer, Stenger, Konecky, & Carter, 2003) or capacity utilization of the cognitive system (Just & Carpenter, 1993; Just, Carpenter, & Miyake, 2003). 

Importantly, pupil dilations related to cognitive processing are thought to result from an inhibitory effect on the parasympathetic oculomotor complex by release of norepinephrine (NE) from the locus coeruleus (LC; Wilhelm, Wilhelm, & Lüdtke, 1999). The LC is a small brainstem nucleus located in the rostral pons that sends its noradrenergic projections to virtually all brain regions (except the basal ganglia), with particularly dense projections to areas known to be important in attentional processing, such as the parietal cortex, the pulvinar nucleus of the thalamus, and the superior colliculus (SC; Bouret & Sara, 2005; Foote & Morrison, 1987; Schneider & Kastner, 2009). The LC-NE system is involved in regulating cortical arousal levels associated with sleep/waking behavioral states (Berridge, 2008), and within the waking state regulation of arousal by the LC-NE system, it interacts with top-down cortical systems involved in attentional control and executive functions (Corbetta, Patel, & Shulman, 2008; Sara & Bouret, 2012). As a neuromodulator, NE may have the distinctive role of controlling the randomness in action selection (Friston, 2009) and the optimization or consolidation of task-related decision processes (Aston-Jones & Cohen, 2005; Einhäuser, Koch, & Carter, 2010). Hence, the LC–NE-based modulatory mechanisms can also help establish shifts in awareness of either external or internal stimuli or, in other words, event boundaries in consciousness where one event becomes relevant and consolidates in awareness, whereas alternative events are consigned to a negligible status (Einhäuser, Stout, Koch, & Carter, 2008; Kurby & Zacks, 2008; Laeng et al., 2012). Importantly, for the present study, single cell recordings in monkeys (Joshi, Kalwani & Gold, 2013; Rajkowski, Kubiak, & Aston-Jones, 1993) and pharmacological studies in humans (Koss, 1986; Phillips, Szabadi, & Bradshaw, 2000) suggest a link between activity of the LC-NE system and changes in pupillary diameter, allowing the use of pupillometry to tap task-related changes in attentional states mediated by LC-NE activity (Laeng et al., 2012). 

A recent conceptualization of mental effort relates the concept to the top-down control of attention; that is, rather than being primarily a function of task difficulty, attentional effort results from an interaction between task demands and the performers' current goals and motivation to stay on task, as well as their ability to recover performance following detrimental events (Sarter et al., 2006). In top-down attentional control, or goal driven attention, cognitive factors such as current goals and intentions, prior knowledge, and expectations are thought to interact with incoming sensory signals, biasing the competition between objects for access to working memory (Corbetta & Shulman, 2002; Desimone & Duncan, 1995). Corbetta et al. (2008) have described two independent but interacting attentional systems in the human brain: a dorsal frontoparietal system, involved in top-down or goal-driven attention, and a ventral frontoparietal system, involved in the detection of behaviorally relevant stimuli and the control of attentional shifts (e.g., interrupting and resetting). The interaction between these two attentional systems and the functional integration of the attentional system as a whole have been suggested to depend on noradrenergic modulation by the LC-NE system (Corbetta et al., 2008; Coull, Büchel, Friston, & Frith, 1999; Sara, 2009; Wen, Yao, Liu, & Ding, 2012). In addition, recent theories of LC function (Aston-Jones & Cohen, 2005) suggest that cortical modulation by the LC-NE system, mediating focused attention and task engagement, may be associated with a specific form of LC activity; in what is termed the LC-phasic mode, the LC responds with phasic bursts of activity to behaviorally relevant stimuli, thought to act as a temporal attentional filter. 

A functional MRI (fMRI) study by Raizada and Poldrack (2008) found activations in a brainstem area corresponding to the location of the LC that correlated with attentional demands, suggesting that LC phasic activity is involved in signaling attentional challenge and resource allocation in humans. It is therefore of great interest to investigate whether pupil size changes can signal activity of cortical networks involved in goal-directed attention. Most relevant for the present study, several fMRI studies of attentive tracking of dynamic visual stimuli have revealed a linear increase in activity of several cortical areas with tracking load. The paradigm is generally known as Multiple Object Tracking (MOT; Figure 1) and requires the participant to split or divide their attention into multiple, separate foci of attention (Pylyshyn & Storm, 1988; Cavanagh & Alvarez, 2005; Scholl, 2009; however, see Jans, Peters, & De Weerd, 2010, for a broader discussion on multifocal attention). The participant's task is to covertly track, by maintaining central fixation of gaze, a set of targets during a tracking interval, and then at the end of the interval indicate which objects in the display were the target objects. By parametrically varying the number of target objects, or load, this task allows a precise control on the required degree of attentional effort. Specifically, these fMRI studies of multiple object tracking have shown activation in core regions of the dorsal frontoparietal attention network (Corbetta et al., 2008), which includes the frontal eye fields (FEF), anterior and posterior intraparietal sulcus (aIPS and pIPS, respectively), and the superior parietal lobule (SPL; Culham et al., 1998; Culham, Cavanagh, & Kanwisher, 2001; Howe, Horowitz, Morocz, Wolfe, & Livingstone, 2009; Jovicich et al., 2001). However, activations in the LC in humans during changes in mental effort, operationalized as changes in pupil diameter, have not so far been investigated with fMRI. 

Figure 1

 

The MOT task. The task starts with a display presenting the objects (i.e., blue colored disks). Then a subset of the disks turns red (in tracking trials). All colors return to blue before they starting moving randomly inside the tracking area (gray). At the end of the tracking period, the participants respond to a probe (i.e., one of the disks changes to green), where the participant answers “yes” or “no” on whether the probed object was one of the original target objects (fMRI session). In the eye-tracking session a full report was required where the participants selected all of the tracked objects using a mouse cursor.

Thus, in the present study, we investigate mental effort as expressed by both changes in brain activity and pupil size during attentive tracking of a variable number of dynamic visual stimuli. We hypothesize that during MOT the pupil diameter will reflect changes in individuals' resource allocation as the number of objects to be tracked increases. Therefore, in the present study, we will model the brain activity according to two separate but connected measures of intensity during MOT: (a) attentional load, which is operationalized by the number of objects to be tracked and thus reflects external task demands, and (b) attentional effort, a latent psychological construct referring to individual differences in attentional load relative to total resource capacity operationalized by the pupillary response (Ahern & Beatty, 1979; 1981; Kahneman, 1973). Given the inherent flexibility of multifocal attention (Franconeri, Alvarez, & Cavanagh, 2013) and the fact that the attentional effort that one invests in a task is highly variable between different individuals (Oksama & Hyönä, 2004; Sarter et al., 2006), we expect that an individual's pupillary response will be a better predictor of LC's activity than load or the physical number of objects to be tracked. To investigate the proposed relationship between pupil dilations, the brainstem LC-NE system, and cortical attentional systems, the same participants also performed the MOT task while in a magnetic resonance imaging (MRI) scanner.We expect individual variations in pupil dilation to correlate with BOLD activity in the LC, as well as in target sites of the LC-NE system in the cortical attention system. Revealing such an extended network in the human brain during attentive tracking and in relation to changes in externally observable pupillary responses would strongly support current cognitive neuroscience accounts of mental effort, which so far have received piecemeal support by separate studies using the methods of neuroimaging or pupillometry independently of one another. 

Materials and methods

Participants

Seventeen subjects (ages 22–48, five males), who were a subsample of the Norwegian Cognitive Neurogenetics sample (Espeseth et al., 2012), gave their informed consent to participate in the study, which had been approved by the Regional Committee for Medical and Health Research Ethics (South-East Norway). None of the participants had present or previous neurological or psychiatric conditions. All participants performed two sessions of the experiment, which occurred in different days: In the first session pupil diameter was recorded by use of an eye tracker while subjects performed a MOT task (Figure 1). In the second session of the study the same participants performed the MOT task in the MR scanner. The average time between the pupillometry and fMRI sessions was 77 days (SD = 46 days). In addition, nine of the previous participants also performed a follow-up pupillometry session, which occurred approximately four years after the first two sessions, in order to investigate stability of individual pupil dilations across time and between versions of the MOT task used in the eye-tracking and MR session. 

Eye-tracking sessions

Stimulus presentation and MOT task

Stimuli were generated using MATLAB® (MathWorks, Natick, MA) and the Psychophysics Toolbox extensions (Brainard, 1997; Pelli, 1997) and presented on a calibrated 21-in. Eizo CRT Monitor (Eizo, Hakusan, Japan) at a resolution of 1280 × 960 at 60 Hz. Participants were seated 90 cm from the screen, and movements where minimized by use of a chin and headrest. Illumination of the room was kept constant for all participants, and object colors used were isoluminant (9 cd/m², measured using Spyder 4, Datacolor, Lawrenceville, NJ). The tracking area was a gray square subtending 17° × 17° visual angle. Each trial started with the presentation of a white central fixation point (0.2° visual angle) and 10 blue objects. The objects were circular disks with a diameter of 0.7° visual angle. After 2.5 s a subset of the objects (or none in passive viewing trials) changed color to red for another 2.5 s, designating them as targets to be tracked, before returning to blue. After a brief interval of 0.5 s, the objects started moving in an unpredictable fashion inside the tracking area. Movement velocity was 5.5°/s, and objects changed directions when object edges were closer than 1° of each other or the edge of the tracking area, and also changed trajectory at random (in any direction) at an average of one change per second to avoid predictable trajectories of movement. Participants were instructed to fixate and track the objects covertly. To further limit unintentional eye movements, the objects changed direction whenever they were within 2° vicinity of the central fixation point. The number of designated targets was zero (Passive Viewing), two, three, four, or five (Load 2–5). A tracking condition of a single object was not included, as we specifically wanted to investigate mental effort involved in divided attentional foci. After 10 s all objects' movements stopped, and the participant indicated which objects they had been tracking using the mouse cursor (Full report). All participants completed four runs of the experiment, each run consisting of five trials per tracking condition including passive viewing. The order of the load conditions was pseudorandomized to make sure that all load conditions had been run before any of them was repeated. 

Follow-up pupillometry session

The follow-up session used the identical task as described above, but in addition we collected pupillometry data for the version of the task originally used in the fMRI session (which differed in some aspects compared to the pupillometry version, as described for the fMRI session). The follow-up session was performed in the same eye-tracking lab and with the same equipment and display parameters as for the first pupillometry session. A subsample of nine of the original sample of participants who had performed four sessions for each of the experiments volunteered to return to the lab after a time lag of several years. 

Data acquisition, preprocessing, and analysis

The pupillometry session was performed in an eye-tracking lab, where the participant's left eye pupil diameter was measured using an iView X R.E.D. Hi-Speed eye-tracking system (SensoMotoric Instruments, SMI, Teltow, Germany) at a sampling rate of 240 Hz, and data recorded using the iView X Software (SMI, Teltow, Germany). Pupillometry data were preprocessed using a custom made MATLAB script to remove artifacts in the time series related to eye blinks: Data points with physiologically unlikely pupil sizes (smaller than 2 mm or larger than 7 mm,) together with the neighboring data points (the preceding and following 50 ms) were removed. Also, samples more than 2.5 SDs from the mean pupil size within a trial were removed from the time series, before the vertical and horizontal pupil measurements were averaged creating a single time series. Trials with less than 50% of the data remaining after removal of outliers were not included for further analysis. Resulting gaps in the data were replaced by linear interpolation, before the resulting time series were smoothed using a robust Loess algorithm (smoothing parameter = 0.1) and then resampled in time bins of 100 ms. Baseline pupil diameter was calculated as the average pupil size during the 200 ms preceding each tracking onset, (all objects present and blue) and was subtracted from the time series of the tracking period. Trials within each of the load conditions for each participant were then averaged, before group average time series and standard errors for each condition were calculated. The average pupil dilation from baseline for each load condition was then calculated from the time period 3–8 s after target onset. This period was selected based on the observation that pupil dilation reaches an asymptote between 2 and 3 s after tracking onset. Also, because of the fixed, and therefore easily predictable length of the tracking periods, pupillary dilations related to preparatory processes to the end of the tracking epoch and the forthcoming response intervals would be expected towards the end of the tracking period (Richer & Beatty, 1985). 

fMRI session

Stimulus generation and presentation

Stimuli were generated using MATLAB (MathWorks, Natick, MA) and the Psychophysics Toolbox extensions (Brainard, 1997; Pelli, 1997), and presented on a calibrated MR-compatible LCD screen (NNL LCD Monitor®, NordicNeuroLab, Bergen, Norway), placed behind the scanner bore. Participants viewed the screen through a mirror mounted on the scanner coil, with an effective viewing distance of 1.2 m. Screen resolution was set to 1920 × 1080 at 60 Hz, and the field of view measured 32° visual angle. Stimulus display and object size/movement parameters were identical to the Pupillometry version (matched in visual degrees). Participants produced their responses using a MR-compatible subject response collection system (ResponseGrip®, NordicNeuroLab, Bergen, Norway) and both response accuracy and response times (RT) were recorded. A trigger pulse from the scanner synchronized the onset of each tracking trial of the experiment to the beginning of the acquisition of an fMRI volume. 

MOT task

The stimuli and task used were essentially the same as in the eye-tracking session, with a few changes: The total number of objects in the display was kept constant at 12, and the tracking period lasted 12 s. At the end of the tracking period all objects' movements stopped, and after a brief interval (1 s), one of the objects was highlighted in green (Probe one), and the participant responded, as quickly but as accurately as possible, whether the probed object was one of the targets that was originally selected to be tracked. Response accuracy and RT were recorded for each trial. The probability that the probed object was one of the targets was 50% for all tracking conditions. The response window was 2.5 s, after which the trial ended and followed by a 4-s inter-trial interval (ITI). One run consisted of five blocks per condition (passive viewing and load 2–5), and the participants were instructed to fixate during both the passive viewing and tracking periods and to track using attention without moving gaze. The load-conditions were semirandomized so that all load conditions had been run before starting a new random sequence of all load conditions. A rest period of 11 s always followed after running through five blocks. Except for the target designation period, visual stimulation was identical across conditions, and only tracking load differed during the 12-s tracking interval. Each participant performed three runs of the task. 

Data acquisition

Scanning was performed with a Philips Achieva 3 Tesla MR scanner (Philips Medical Systems, Best, The Netherlands), equipped with an eight-channel Philips SENSE head coil. Functional data were collected using a BOLD-sensitive T2* weighted echo-planar imaging sequence (36 slices, no gap; repetition time [TR], 2 s; echo time [TE], 30 ms; flip-angle, 80°; voxel size, 3 × 3 × 3; field of view [FOV], 240 × 240 mm; interleaved acquisition). The slices were oriented to cover the whole cortex, cerebellum and the brainstem's pons. To avoid T1 saturation effects, five dummy scans were collected at the start of each fMRI run. Each run produced 365 volumes. Anatomical T1-weighted images consisting of 180 sagittally oriented slices were obtained using a turbo field echo pulse sequence (TR, 6.7 ms; TE, 3.1 ms; flip angle 8°; voxel size 1 × 1.2 × 1.2 mm; FOV, 256 × 256 mm). 

Preprocessing for cortical analysis

Imaging data was pre-processed and analyzed using BrainVoyager QX software (version 2.8, Brain Innovation, Maastricht, The Netherlands). The high-resolution T1 images were skull-stripped and transformed to the Talairach coordinate system (Talairach & Tournoux, 1988). A group average cortical mesh was generated using a surface-based cortex alignment procedure: Each subject's anatomical image was corrected for spatial intensity inhomogeneities, white-/gray-matter boundaries were estimated and bridges removed, before three-dimensional (3-D) meshes of each subject's cortical surface were created. These meshes were then morphed into spheres and aligned to each other using a moving target group averaging approach, before creating a group-average reconstruction of the cortical surface. fMRI data were slice-time corrected using sinc interpolation, and corrected for intrasession movement using a six parameter model, before linear trends and low-frequency components were removed (high-pass filter of 0.0068 Hz). The voxel time courses were then coregistered to their corresponding T1 images, transformed into the Talairach coordinate system, spatially smoothed using a Gaussian kernel (full width at half maximum [FWHM] 5 mm) and z-normalized using baseline segments of the time series. 

Preprocessing for brainstem analysis

In order to investigate activations in the brainstem and specifically the LC, we performed a brainstem specific analysis following the method described in Keren, Lozar, Harris, Morgan, and Eckert (2009). First each individual's brainstem was cropped and aligned to each other using VMR-VMR (3D anatomical) coregistration in BrainVoyager, before exporting them, along with each participants' unsmoothed functional data, to the NIFTI-file format using functions from the NeuroElf toolbox (v. 0.9c) for MATLAB, for further processing in the Statistical Parametric Mapping (SPM8) software. The aligned brainstems were averaged, creating a study specific brainstem template, before each participant's cropped brainstem was normalized a binarized version of the study-specific template, conforming them to the same global shape and size. The resulting normalized brainstems were then averaged, and this average study-specific normalized image was normalized to a cropped MNI152 T1-template. The resulting estimation parameters were then used to transform each individual's functional data set to MNI-space. For one of the participant's the fMRI bounding box did not cover the entire pons, and the dataset was therefore excluded from the brainstem specific analysis, resulting in 16 subjects for the brain-stem specific analysis. The cropped MNI152-brainstem template, the MNI-normalized functional data, and the binarized two standard deviation (2STD) LC-MNI mask described in Keren et al. (2009) were then imported back into BrainVoyager for statistical analysis and visualization. 

fMRI analysis

Since earlier studies have modeled task difficulty in the MOT paradigm using a linear parametric model (Culham et al., 2001; Jovicich et al., 2001), we wanted to investigate whether changes in pupil diameter could explain variation in BOLD signal beyond a linear increase as a function of number of targets. A whole-brain random-effects analysis was conducted using a GLM with four regressors of interest: (a) passive viewing, modeling the tracking period in trials with zero target objects; (b) active tracking, modeling the tracking period in trials in which participants actively tracked two, three, four, or five targets; (c) linear parametric, modeling a linear increase in activity as a function of the number of target objects that were tracked in a trial (excluding the passive viewing condition); (d) pupil size, modeling a parametric change in activity as a function of the change in the participant's pupil size in each of the active tracking conditions (load 2–5). The pupil size regressor contained each participant's average change in baseline corrected pupil size (mm) from the passive viewing condition, as measured in the eye tracking session. Both parametric regressors were mean-centered around zero. Weights were −1.5, −0.5, 0.5, and 1.5 for the linear arametric regressor, while the weights for the pupil regressor were based on the average individual pupil response for each of the tracking conditions, with the mean of the four values subtracted. Since the linear and parametric regressor are expected to show a high correlation, the column representing the pupil regressor in the design matrix was orthogonalized with respect to the parametric regressor, using the function “orthvec” from the NeuroElf toolbox (v. 0.9c). The pupil regressor thus represents the individual pupil dilations (as a measure of individual differences in attentional effort), when the effect task difficulty (number of objects tracked) is factored out. All predictors were convolved with a two-gamma canonical haemodynamic response function (HRF) and intrinsic autocorrelations modeled using an autoregressive model of order 2 (AR(2)) model. Three t-maps were then calculated: one for the contrast tracking (load 2–5) > passive viewing, and two representing the main effects of the linear parametric regressor and the pupil parametric regressor, respectively. The statistical maps were corrected for multiple comparisons by estimating empirical cluster-level thresholds using Alphasim (Ward, 2000), as implemented in the NeuroElf toolbox (http://neuroelf.net) for MATLAB (whole brain analysis: cluster-defining threshold of p = 0.05, the p = 0.05 familywise error (FWE)-corrected critical cluster size: 53 voxels; brainstem specific analysis: cluster defining threshold of p = 0.05, the p = 0.05 FWE-corrected critical cluster size: 10 voxels). 

Results

Pupillometry session

Behavior

Figure 2 shows the proportion of correctly identified targets across the active tracking conditions. A repeated-measures ANOVA revealed a significant effect of tracking load on proportion of correctly identified targets, F(1.986, 31.782) = 27.241, p < 0.01. There was a significant linear trend, F(1, 16) = 42.499, p < 0.01, indicating a proportional decrease in response accuracy as tracking load increased. 

Figure 2

 

Tracking performance in the eye-tracking session of the MOT task. Participants gave a full report (marking all tracked objects at the end of a tracking trial). Shown here is the proportion of correctly identified targets over number of targets being tracked (from two to five). Error bars represent SEM.

Pupil

Figure 3 shows the time series for the group average pupil diameter change (in mm) during the tracking interval. One can clearly see the effect of tracking on the eye pupil, where the diameter gradually increases in size and maintains a constant magnitude during the interval, whereas during the passive viewing trials the pupil gradually decreases in size. Figure 4 shows the average pupil dilations for the interval 3-8 s of the tracking period. A repeated-measures ANOVA with Greenhouse-Geisser correction revealed a significant effect of number of targets on mean pupil size change, F(2.045, 32.71) = 5.91, p < 0.01. There was a significant linear trend, F(1, 16) = 9.261, p < 0.01, indicating a proportional increase in pupil size as tracking load increased. 

Figure 3

 

Pupil time series, eye-tracking session. Group averages of the pupil time series during the eye-tracking session of MOT. x-axis represents the tracking period in seconds, y-axis the pupil size in mm. The pupil was clearly sensitive to the parametric changes in attentional load. Shaded area represents SEM.

Figure 4

 

Average pupil dilations in mm from baseline across tracking. The baseline was calculated as the 200-ms epoch preceding each tracking onset. Error bars represent SEM.

fMRI session

Behavior

Figure 5 shows the mean response accuracy across the load conditions. There was a significant effect of tracking load on accuracy, F(3, 48) = 16.442, p > 0.01, and a significant linear trend, F(1, 16) = 29.262, p < 0.01, indicating a proportional decrease in accuracy as tracking load increases. 

Figure 5

 

Tracking performance in the eye-tracking session of the MOT task. The participants gave a partial report; that is, one object was probed and the participants answered “yes” or “no” on whether the probed object was one of the targets in a trial. In each trial there was a 50% likelihood of the probe being a target. Error bars represent SEM.

fMRI

Contrasting tracking (Load 2–5) over passive viewing (Load 0), we expected to replicate earlier MOT-fMRI studies, revealing extended networks involved in visual attention during active tracking. Following the naming convention of earlier MOT-fMRI studies (Howe et al., 2009; Jovicich et al., 2001), contrasting active tracking to passive viewing (Figure 6; Table 1) did reveal large activations in areas belonging to the dorsal attention network and the visual perceptual system, including the supplementary motor area (SMA), the superior and inferior precentral sulcus and the FEF at the junction of the precentral sulcus and the superior frontal sulcus, the medial frontal sulcus, anterior parts of the insular cortex, the right middle frontal gyrus (MFG), the postcentral sulcus, the transverse parietal sulcus (TranPS), the SPL, aIPS and pIPS, the lateral occipital cortex and middle temporal cortex including the human motion sensitive area (MT+), and the right calcarine sulcus. Subcortical activations are seen in the brainstem reticular formation and midbrain tectum, and also the thalamus including the pulvinar nucleus, the cerebellum and the basal ganglia. Also, several negative activations were seen in areas associated to the brains default network: in the anterior cingulate cortex (ACC), the bilateral ventral medial prefrontal cortex (vmPFC), and orbitofrontal cortex (OFC), the superior and inferior frontal gyrus, the paracentral lobule, the posterior insular cortex, the posterior cingulate cortex (PCC), inferior parietal lobule, anterior parts of the superior temporal sulcus, the hippocampus and parahippocampal cortex, and the posterior parts of the inferior occipital gyrus. 

Figure 6

 

t-map showing tracking related activations (trackings > passive viewing). The map was corrected for multiple comparisons using Alphasim (p < 0.05, k = 53) and projected on the group cortical mesh. Numeric labels represent: (1) middle frontal gyrus (MFG); (2) frontal eye fields (FEF); (3) anterior intraparietal sulcus (aIPS); (4) superior parietal lobule (SPL); (5) posterior intraparietal sulcus (pIPS); (6) MT+; (7) transverse parietal sulcus (TransPS); and (8) supplementary motor area (SMA).

Table 1

 

Summary of the fMRI results. Notes: The columns represent the contrast tracking > passive viewing (Track), the main effect of the linear parametric regressor (Load), and the pupil regressor (Pupil). Positive activations are marked with a plus sign (+) and negative with a minus sign (−). Lateralized activations are represented with L and R, for left and right lateralized activations, respectively. The remaining activations are bilateral.

	Tracking	Load	Pupil
Frontal
Anterior cingulate cortex	−	−
Supplementary motor area	+	+
Inferior precentral gyrus	+	+	+ (L)
Precentral sulcus	+	+	+
Frontal eye fields	+	+	+
Anterior insular cortex	+	+
Middle frontal gyrus	+	+
Ventral medial prefrontal cortex/ orbitofrontal cortex	+	−
Superior frontal gyrus	−	−	−
Inferior frontal gyrus	−	− (R)
Medial frontal gyrus	+	+
Posterior insular cortex	−	−
Parietal
Paracentral lobule	−	−
Postcentral sulcus	+
Transverse parietal sulcus	+	+ (L)	+ (R)
Anterior intraparietal sulcus	+	+
Superior parietal lobule	+	+	+
Inferior parietal lobule	−	−
Posterior intraparietal sulcus	+	+	+ (L)
Posterior cingulate cortex	−	−	− (L)
Temporal/occipitial
Superior temporal gyrus	−	−
Hippocampus/parahippocampal gyrus	−	−	− (L)
Superior temporal sulcus	−	−
Inferior occpitial gyrus	−
MT+	+	+	+ (R)
Lateral occipital cortex	+	+	+
Calcarine sulcus	+ (R)
Subcortical
Basal ganglia	+	+ (L)
Thalamus	+	+	+ (R)
Superior colliculus	+	+	+ (R)
Locus coeruleus	+	+ (R)	+
Cerebellum	+	+	+

The main effect of the linear parametric regressor (Figure 7A; Table 1), reflecting attentional load as operationalized by the number of tracked targets, revealed a parametric increase with number of tracked objects bilaterally in the FEF, the SMA, the medial frontal gyrus, anterior insular cortex, inferior parts of the precentral gyrus, the MFG, and inferior parts of the precentral sulcus, the aIPS, pIPS, and SPL, the left TranPS, the MT+, the right thalamus, bilateral SC, as well as the right LC (Figure 7B). A parametric decrease with number of tracked objects was seen bilaterally in the ACC and the anterior parts of the superior frontal gyrus, the right vmPFC and OFC, anterior parts of the superior frontal gyrus, bilateral inferior parietal cortex, posterior cingulate cortex, inferior parietal lobule, parahippocampal gyri, and anterior parts of the superior temporal sulcus. 

Figure 7

 

t-maps representing areas showing a linear increase in activation with tracking load (linear parametric). The maps were corrected for multiple comparisons using Alphasim (cortical: p < 0.05, k = 53; brainstem p < 0.05, k = 10). (A) Linear parametric t-map projected on the group cortical mesh. Numeric labels represent: (1) MFG, (2) FEF, (3) aIPS, (4) SPL, (5) pIPS, (6) MT+, (7) TransPS, and (8) SMA. (B) Linear parametric t-map from the brainstem specific analysis shown on a cropped version of the MNI152 template. The LC mask from Keren et al. (2009) is shown in blue.

Figure 8 shows the main effect of the pupil regressor. Since the pupil and the linear regressor showed a high degree of collinearity, the pupil regressor was orthogonalized with respect to the linear parametric regressor. This forces the shared variance between them into the linear parametric regressor, and hence the results shown reflect the unique variance explained by pupil size changes. This regressor revealed a positive relationship with pupil size change bilaterally in the FEF, superior parts of the precentral sulcus, the SPL and lateral occipital cortex, left lateralized activations in the inferior parts of the precentral gyrus and pIPS and right lateralized activations in the TranPS and MT+. Subcortical activations were seen in the right thalamus, the right SC, bilaterally in the cerebellum, as well as in the LC (Figure 8B). A negative relationship was seen in the medial parts of the superior frontal gyrus, the left posterior cingulate cortex and the left parahippocampal gyrus. 

Figure 8

 

t-map showing pupil-related activations. The pupil size predictor was orthogonalized to the linear parametric predictor, forcing shared variance between the two regressors into the linear parametric regressor. The maps were corrected for multiple comparisons using Alphasim (cortical: p < 0.05, k = 53; brainstem p < 0.05, k = 10). (A) Pupil t-map projected on the group average cortical mesh. Numeric labels represent: (1) MFG, (2) FEF, (3) aIPS, (4) SPL, (5) pIPS, (6) MT+, (7) TransPS, and (8) SMA. (B) Pupil t-map from the brainstem specific analysis shown on a cropped version of the MNI152 template. The LC mask from Keren et al. (2009) is shown in blue.

Follow-up pupillometry session

The fMRI analysis used individual pupil dilations collected in a separate session to model BOLD activity in the fMRI session of the MOT experiment. In addition to being collected in different sessions, the MOT display also differed in the number of distractor objects and the length of the tracking period, but in every other respect the two tasks were the same. As studies show that individual differences in attentional capacity are correlated with pupil dilations within given task difficulty levels (Ahern & Beatty, 1979, 1981; Piquado et al., 2010), we reasoned that the individual pupil dilations in the MOT task would reflect the relative amount of attentional resources taxed at each tracking load (i.e., attentional effort), and thus reflect a stable characteristic of each individual's cognitive system. If so, these individual differences in task-evoked pupil dilations will show stability over time and across minor differences of the MOT task. To investigate this, we collected pupillometry data in a follow-up session for nine of the original participants, approximately four years after the previous sessions. Participants performed both the MOT task that was used in the first pupillometry session as well as the version originally used in the fMRI session. One of these participants was excluded from the analysis because of blinking artifacts rendering most of the data useless. To investigate stability across time and across experiments, we calculated group-level within-subject correlations, using multiple regressions. This is equivalent to fitting parallel regression lines through each individual's data (Bland & Altman, 1995). Despite the ample time lag, we observed significant correlations between pupillometry Session 2 and pupillometry Session 1 (both pupillometry versions of MOT), R = 0.84, p < 0.001, and between pupillometry Session 2 (pupillometry version) and pupillometry Session 2 (fMRI-version of MOT), R = 0.732, p < 0.001. Further, since we orthogonalized the pupil regressor with respect to the regressor modeling task difficulty (parametric regressor) in the fMRI analysis, we investigated the stability of the nonlinear variance in the pupil dilations across time and experiments, by including tracking load in the regression. We observed moderate and significant correlations both across time, R = 0.38, p = 0.046, and experiments (R = 0.43, p = 0.024). Results for the regression between the different versions of the experiment is shown in Figure 9. 

Figure 9

 

(A) A subset of the participants performed a second session of the experiment, in order to assess stability in individual pupil dilations across the different MOT versions (for eyetracking [ET] & fMRI) used in the first session. The x-axis represents pupil dilations in the ET version of the MOT task, while the y-axis represents pupil dilations in the fMRI version. Colors indicate the participants. The parallel lines represent the group average regression line fitted through each participant's data point, thereby regressing out the mean of each participant's pupillary response. (B) The second panel shows the residual plot when task difficulty (tracking load) is regressed out.

Discussion

As expected, the pupil diameter increased with the number of objects to be tracked, indicating an increase in mental effort, or allocation of attentional resources, as task demands increased. In addition, we confirmed previous findings showing that specific cortical areas are involved in object tracking, but we also extended them by revealing an involvement of the brainstem, including the putative LC, in attentive tracking. To our knowledge this is the first study using pupil dilations as an index of attentional effort in a MOT task. This allowed us to model individual differences in attentional effort using pupil dilations across task loads as a parameter in a parallel fMRI study. Indeed, we found that activity in the LC, and also in FEF and SPL, correlated with attentional effort as indexed by pupil size changes across tracking load levels. Ideally one should record both pupil diameters and BOLD activity at the same time, as this would better capture moment-to-moment changes in attentional effort and enable a more precise modeling of the relationship between the pupillary response and LC activity. A recent study (Murphy, O'Connell, O'Sullivan, Robertson, & Balsters, 2014) used this method and found that BOLD activity in a pontine cluster overlapping the LC correlated with a concurrently recorded continuous pupil measure during both rest and while performing a two-stimulus odd-ball task. The current study allows us to extend these findings also to experimental designs where the two types of measurements are made at separate time points. The present experiment provides evidence that the pupillary responses to the MOT task can be remarkably stable over a lag of several years in the same individuals, and that pupillary responses measured in one session can be highly predictive of pupillary responses in another session. In turn, this confirms that pupillary changes can provide a reliable index of individual trait differences in mental effort. Most important, in the present study we show for the first time that such a LC-activation correlated to the pupillary changes is systematically modulated by cognitive load. 

We reasoned that MOT is an optimal paradigm for the study of mental effort or the intensive aspect of attention (Kahneman, 1973), since by parametrically increasing the number of objects to be tracked, the task allows in a clear-cut manner the investigation of load-dependent activity in the brain's attentional system. Previous studies have indicated load-dependent activity within several areas of the dorsal attentional system (Culham et al., 2001; Jovicich et al., 2001). By including each individual's mean pupil dilations across tracking loads, recorded in a separate eye-tracking session of the study, as a measure of individual variability in attentional effort in the fMRI session, we were able to model two separate but connected measures of intensity in MOT: attentional load, which is operationalized by the number of objects to be tracked and thus reflects external task demands, and attentional effort, a latent psychological construct referring to individual differences in attentional load relative to total resource capacity (Ahern & Beatty, 1979, 1981). Also, the fact that individual differences in pupil dilations show a high degree of stability over an extended period of time (about four years) lends support to our interpretation that pupil dilations can be used as a reliable measure of such individual differences in attentional effort which can be used in separate measurement sessions. These two measures are of course expected to correlate: As attentional load increases, so does the attentional effort required to successfully perform the task, and this is reflected as an increase in pupil size. As we predicted, this relationship holds true for the MOT task (Figures 3 and 4) since, at the group level, the pupil increases with tracking load. Thus, to investigate whether individual differences in pupil dilations could explain BOLD activity beyond the linear parametric regressor modeling tracking load, the pupil regressor was orthogonalized with respect to the former. Our results show that individual differences in pupil dilation can explain variance in the BOLD data beyond what is explained by modeling the number of objects tracked and, in line with our initial hypothesis, they also predict activity in the LC. 

Finding activations in the human LC with fMRI that are related to the individuals' pupillary responses may seem remarkable since the spatial resolution of fMRI makes it difficult to distinguish activation stemming from brainstem nuclei. Thus, any interpretation of activity within the brainstem must be based on other supportive evidence before concluding that it refers to a specific nucleus (Raizada & Poldrack, 2008). In fact, the whole LC contains in the adult human only 22,000 to 51,000 neurons (Mouton, Pakkenberg, Gundersen, & Price, 1994). Nevertheless, the brainstem cluster we observed in our MR measurements closely matches the anatomical location of the LC as shown in study mapping the LC using high-resolution MRI (Keren et al., 2009). Moreover, the fact that pupillary dilations are thought to reflect LC-NE activity during attentional and cognitive load (Laeng et al., 2012) lends support to our interpretation that the pupil related activations in the brainstem originate from the LC. Further, the pupil-related thalamus activation in our study was lateralized to the right side. A denser concentration of norepinephrine in the right compared to the left thalamus in humans has been reported (Corbetta & Shulman, 2002), further supporting our interpretation implicating the LC-NE system in attentional resource allocating during MOT. Even so, our identification of the LC, as in previous fMRI-studies reporting LC activity (Murphy et al., 2014; Raizada & Poldrack, 2008), is tentative and must be interpreted with caution. Also, the LC is located close to the pontine raphe nuclei (Parvizi & Damasio, 2003), and in the monkey, neurons in the rostral poles of the SC and omnipause neurons (OPNs) in the nuclei raphe interpositus and nuclei raphe pontis can display tonic activity during fixation behavior, preventing the execution of saccades (Büttner-Ennever, Cohen, Pause, & Fries, 1988; Everling, Paré, Dorris, Munoz, & Pare, 1998; Munoz & Wurtz, 1993). OPNs have been hypothesized to be modulated by arousal levels (Di Stasi, Catena, Cañas, Macknik, & Martinez-Conde, 2013), and as our task requires maintaining fixation, arousal dependent and thereby pupil-related activations in the raphe nuclei could be consistent with our observed brainstem cluster. However, as arousal is regulated by the LC-NE system (Berridge, 2008), it seems unlikely that we would find arousal dependent activity in the raphe nuclei without the involvement of the LC. 

The MOT paradigm powerfully demonstrates how visual attention can be split between several regions or objects in the visual field. Moreover, the task reveals the limitations of our visual attentional resources, as increasing demands due to increasing division of resources lead to decreases in performance. The limiting factor in the splitting of visual attention has been suggested to stem from the low-level architecture of the visual system (i.e., according to slot models: Franconeri et al., 2013) one of which is Pylyshyn's (1989) FINST (FINgers of INSTantiation) model, where a limited number of spatial indexes are suggested to “stick” to the tracked objects. However, recent studies suggest that the limitation on tracking is not the number of objects per se, but instead stems from a more flexible allocation of attentional resources, primarily dependent on object crowding (Franconeri, Lin, Pylyshyn, Fisher, & Enns, 2008). Thus, even tracking one object may exceed the available resources at high enough movement velocity, while up to eight objects may be tracked at slower speeds (Alvarez & Franconeri, 2007), depending on demands of representational fidelity to perform the task at hand, and the degree to which object representations interfere with each other. Importantly, pupillary responses have been shown to reliably reflect the intensity of mental operations and specifically the allocation of attention across a range of different tasks and in a continuous manner that reflects the fluidity of changes within the attentional and executive networks (Gilzenrat, Nieuwenhuis, Jepma, & Cohen, 2010; Laeng et al., 2012; Wierda, van Rijn, Taatgen, & Martens, 2012). Indeed, Scholl (2009) has suggested that there might not be anything else to tracking than the deployment of visual attention, describing the MOT task as “tracking in the present”: All one needs to know to identify an object as a target is that it was a target the moment before and there is no need to keep a record of movement for individual objects in the display. By revealing an increase in pupil size with the number of objects tracked, the present results are consistent with the view that tracking involves a flexible allocation of a limited attentional resource in MOT. Further, given the proposed relationship between pupil diameter and LC-NE activity (Koss, 1986; Phillips et al., 2000; Rajkowski et al., 1993), our results suggest that the LC-NE system is involved in regulating attentional resources during multiple object tracking. Since our task involved a continuous load on attention, it is not possible to differentiate between LC tonic state activity and phasic bursts from the pupil data, however given the evidence linking the LC-phasic mode to task-engaged behavioral states (Aston-Jones & Cohen, 2005), we interpret our results as reflecting a sustained LC-phasic mode activity as long as task demands are upheld. 

Further supporting the involvement of the LC-NE system in regulating attentional resources, and also supporting a link between pupil dilations and LC activity in humans, was the presence of pupil related activations in the brainstem reticular formation, including the putative location of the LC. Also, a recent study has shown that stimulation of the monkey SC caused pupil dilations suggesting SC activity as a possible second mediator of the observed correlations between pupil dilations and attentional load (Wang, Boehnke, White, & Munoz, 2012). Indeed in the present results we did observe pupil related activity also in the SC. Both the LC and the SC have been linked to goal-driven attentional processes, where the SC contains a topographic map of the contralateral visual hemifield, possibly implicating it in covert spatial shifts of attention (Lovejoy & Krauzlis, 2010; Wang et al., 2012), while the LC is proposed to mediate cognitive and attentional processing via arousal via NE-modulation of cortical attentional systems (Aston-Jones & Cohen, 2005; Sara, 2009). Recent studies have shown that the LC-NE system is especially important in situations demanding quick shifts of attentional focus, where the phasic bursts of the LC is proposed to act as a network reset signal (Bouret & Sara, 2005; Sara & Bouret, 2012), facilitating reorganization of attentional networks linking behaviorally relevant stimuli to appropriate motor responses (Corbetta et al., 2008), and in response to novel stimuli or unexpected stimulus-reinforcement contingencies (Sara & Segal, 1991; Vankov, Hervé-Minvielle, & Sara, 1995). However, the LC-NE system also seems to play an important role in sustained attention (Aston-Jones & Cohen, 2005; Sara & Bouret, 2012), and optimal noradrenergic modulation of the prefrontal cortex is important for selective attention and executive function in nonhuman primates (Sara, 2009), and probably also in humans. For example, Coull and colleagues (1999) showed that administration of the α₂ adrenoreceptor agonist clonidine increased the effective connectivity between the LC and frontal and parietal cortex during a visual attention task. Also, a recent study by Raizada and Poldrack (2008) reported LC activation associated with unpredictable demands on attentional resources in an audiovisual task. They also reported areas in the right frontal cortex, as well as the parietal, visual and auditory cortex showing high correlation with LC activity. The role of LC-NE system in attention and cognition may therefore be to facilitate the functional integration of brain areas involved in these tasks (Coull et al., 1999; Sara, 2009). 

The act of tracking versus passively viewing objects also revealed a cortical activation pattern that largely replicated earlier fMRI studies using the MOT task (Culham et al., 2001; Howe et al., 2009; Jovicich et al., 2001). We observed large bilateral frontoparietal activations in the inferior precentral sulcus, the putative location of the FEF, the SMA, the aIPS, the SPL, the TranPS, the pIPS, and also the lateral occipital cortex and MT+. The precentral sulcus including the FEF, the IPS, and SPL are considered core regions of the dorsal attention network, which mediates top-down or goal-driven attention, biasing competition between stimuli competing for attentional resources based on current goals or task set (Corbetta et al., 2008). We also revealed tracking-related activation in the right calcarine sulcus (primary visual cortex/V1). Interestingly, attentional modulation of activity in the primary visual cortex has not been reported in any of the above-mentioned studies on MOT, but it has been shown by previous fMRI studies using other tasks involving spatial attention (Lauritzen, D'Esposito, Heeger, & Silver, 2009; Silver, Ress, & Heeger, 2007; Somers, Dale, Seiffert, & Tootell, 1999). Importantly, a recent study measuring steady state visual evoked potentials (SSVEPs) during multiple object tracking revealed that multifocal attention enhances processing of targets objects in early visual cortex, including V1 (Störmer, Winther, Li, & Andersen, 2013). This finding, together with the results provided by our fMRI results, thereby converges to support for the involvement of early visual cortex in multiple object tracking, as it is assumed by the multifocal attention model (Cavanagh & Alvarez, 2005). However, it remains unclear as to why the activity is lateralized to the right visual cortex in the present study. One possibility is that feedback circuits from the right hemisphere, known to be dominant in attentional control (Corbetta & Shulman, 2011), may be stronger in the primary visual areas within the same hemisphere. 

Our results also showed that core areas of the dorsal attention network, including the FEF and the SPL, correlate with individual measures of attentional effort operationalized as pupil dilations across tracking load. As suggested by Howe et al. (2009) the FEF and SPL may be active in MOT due to suppression of eye movements as subjects are required to fixate, as these areas have been implicated in saccade generation and suppression. The dorsal frontoparietal attention overlap with neural systems for oculomotor control, and subcortically the SC has also been shown to mediate both overt eye movement and covert attention (Awh, Armstrong, & Moore, 2006; Corbetta & Shulman, 2002; Serences & Yantis, 2006). Covert attention involves decoupling the locus of attention from the foveal parts of the visual field, and a recent study showed that the FEF, SPL, and IPS were more active when maintaining attention in the periphery, when compared to maintaining attention foveally, and was also accompanied by activity modulations of extrastriate areas (Kelley, Serences, Giesbrecht, & Yantis, 2008). Topographic maps involved in saccade planning both in the dorsal attentional system and the SC may therefore be responsible also for biasing competition between objects in the visual field when tracking with covert attention in MOT. According to Franconeri and colleagues (Franconeri et al., 2013) it is the cortical representational space of these maps that is the fundamental bottleneck on attentional capacity. As a measure of relative capacity utilization, it is interesting that the changes in pupil diameter correlate with activity in regions containing such topographic maps, while other areas showing load dependent activity such as insula and MFG do not. 

To conclude, we believe that the present results support a specific interpretation of the concept of mental effort: While the capacity aspect of attention may be related to cortical representational space in two-dimensional (2-D) cortical maps, our results suggest, in line with a proposal by Sarter et al. (2006) that the intensity aspect of attention is closely related to the activation of top-down mechanisms, presumably modulating activity in such representational maps based on the current task demands (Roggeman, Fias, & Verguts, 2010). Specifically, Sarter et al. proposed that increases in task performance with increasing attentional effort stems from the activation of top-down mechanisms which serve to optimize information processing, thereby enabling stable performance and recovery from degraded performance when it is detected. Franconeri and colleagues (2013) have proposed that the abstract and often used terms “capacity limitations” and “resources” are meaningful and they can be translated into more concrete concepts of available representational space in 2-D cortical representational maps. In the end, it seems that Kahneman's (1973, 2011) metaphor of the pupil acting just like the gauge outside the house measuring “energy expenditure from the electricity grid” as an index of mental effort, is not at all inconsistent with the cortical real estate metaphor proposed by Franconeri and colleagues (2013). 

Acknowledgments

We would like to thank our research assistants Cecilie Skaftnes, Tommy Sinnes, and Knut Haavard Kløvfjell for their contributions with eyetracking and fMRI data collection. Thomas Espeseth received Research Council of Norway grant 177458/V50. 

Commercial relationships: none. 

Corresponding author: Dag Alnæs. 

E-mail: [e-mail]. 

Address: Department of Psychology, University of Oslo, Oslo, Norway.