Open Access
Article  |   December 2018
The role of implicit perceptual-motor costs in the integration of information across graph and text
Author Affiliations
Journal of Vision December 2018, Vol.18, 16. doi:10.1167/18.13.16
  • Views
  • PDF
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Jason F. Rubinstein, Eileen Kowler; The role of implicit perceptual-motor costs in the integration of information across graph and text. Journal of Vision 2018;18(13):16. doi: 10.1167/18.13.16.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Strategies used to gather visual information are typically viewed as depending solely on the value of information gained from each action. A different approach may be required when actions entail cognitive effort or deliberate control. Integration of information across a graph and text is a resource-intensive task in which decisions to switch between graph and text may take into account the resources required to plan or execute the switches. Participants viewed a graph and text depicting attributes of two fictitious products and were asked to select the preferred product. Graph and text were presented: (1) simultaneously, side by side; (2) sequentially, where the appearance of graph or text was triggered by a button press, or (3) sequentially, where the appearance of graph or text was triggered by a saccade, thus requiring cognitive effort, memory, or controlled processing to access regions out of immediate view. Switches between graph and text were rare during initial readings, consistent with prior observations of perceptual “switch costs.” Switches became more frequent during re-inspections (80% of time). Switches were twice as frequent in the simultaneous condition than in either sequential condition (button press or saccade-contingent), showing the importance of perceptual availability. These results show that strategies used to gather information while reading a graph and text are not based solely on information value, but also on implicit costs of switching, such as effort level, working memory load, or demand on controlled processing. Taking implicit costs into account is important for a complete understanding of strategies used to gather visual information.

Introduction
A fundamental principle governing visuomotor activity is to achieve the desired goal while minimizing the costs associated with planning or executing the actions (Shenhav et al., 2017; Wolpert & Landy, 2012). Costs may be specified explicitly, for example, as rewards or penalties attached to different possible outcomes (Constantino & Daw, 2015; Dean, Wu, & Maloney, 2007; Trommershäuser, Maloney, & Landy 2008; Wolfe, 2013). Often, however, the costs that affect the choice of strategy are implicit. For example, implicit costs may include the time required to complete the task, even when minimization of time is not a requirement (Araujo, Kowler, & Pavel, 2001; Moher & Song, 2014). Costs may also include the demands made on working memory or other limited cognitive resources (Ballard, Hayhoe, & Pelz, 1995; Epelboim & Suppes, 2001; Hayhoe, 2017; Leider & Griffiths, 2017). The present study examines the role of the implicit costs associated with the actions needed to perform a high-level visual task, namely, reading a graph and its accompanying text. 
The importance of implicit costs was recently discussed by Shenhav et al. (2017). They argued that the mental effort required to plan actions or to make decisions is “inherently aversive or costly” (p. 102), where mental effort was defined broadly to include the management or control of limited pools of cognitive resources. According to their view, limits in the capacity for effortful, controlled mental operations lead to preferences for strategies that reduce the expenditure of mental effort, even when such strategies do not lead to measured improvements in the performance of the task. 
Previous studies of the strategies used during visuomotor tasks have supported the view that implicit costs, such as time or effort (mental or physical), are taken into account when selecting the strategy. Moher and Song (2014), for example, studied reaching movements made to a target in the presence of a nontarget. When the separation between the target and the nontarget was small, reaches were initiated earlier and trajectories contained midcourse corrections. Increasing the separation led to the opposite pattern, namely, longer latencies and fewer corrections. Moher and Song (2014) suggested that the strategies took into account that corrective movements would have higher biomechanical costs and take more time when separations were large. Analogous results to these have been obtained in oculomotor tasks that elicit the so-called “center of gravity” saccades, which are short latency saccades that land near the center of a configuration of a target surrounded by one or more nearby distractors (Coëffé & O'Regan, 1987; Findlay, 1982; He & Kowler, 1989; Ottes, Van Gisbergen, & Eggermont, 1985). Coëffé and O'Regan (1987) offered a rational basis for center-of-gravity saccades by showing that targets surrounded by nearby distractors could be localized sooner with a rapidly initiated saccade to the center of the configuration, followed by a correction, than with a single, more accurate, longer-latency saccade (also, Cohen, Schnitzer, Gersch, Singh, & Kowler, 2007; Wu, Kwon, & Kowler, 2010). 
Previous work has also shown that implicit costs in the form of time, mental effort, or demand on limited processing resources influenced the planning of gaze shifts during different types of visual tasks. During visual search, for example, local cues that provided the searcher with useful information about the location or the value of targets were found to be neglected when taking the cue into account would have prolonged fixation times or risked diverting processing resources away from other aspects of the search task (Araujo et al., 2001; Hooge & Erkelens, 1998, 1999; Navalpakkam, Koch, Rangel, & Perona, 2010; Wu & Kowler, 2013). By contrast, cues woven into the semantic content of display, which presumably can be noticed without adding to processing time or processing load, influenced scanning strategies (Koehler & Eckstein, 2017; Malcolm & Henderson, 2010; Neider & Zelinsky, 2006; Torralba, Oliva, Castelhano, & Henderson, 2006; Wu, Wick, & Pomplun, 2014). Other studies showed that during visual problem-solving tasks, increasing the time or effort required to plan or carry out the gaze shifts, either by increasing the distance between critical locations (Ballard et al., 1995), or by creating long artificial delays (750 ms) between the arrival of gaze at a location and the appearance of visual information (Kibbe & Kowler, 2011), led to less reliance on eye movements, and a greater reliance on memory to retrieve the contents of previously examined locations. 
The studies reviewed above suggest that understanding strategies of visual information-gathering requires considering the implicit costs of planning or executing the actions. In keeping with previous arguments (e.g., Kool, McGuire, Rosen, & Botvinick, 2010; Monsell, 2003; Shenhav et al., 2017; Wolpert & Landy, 2012;), we define implicit costs broadly to include time or motor effort, as well as the use of controlled, rather than automatic, processing. The current understanding of the role of implicit costs in information gathering is limited because many prior studies of the strategies used to gather visual information from visual displays used actions (gaze shifts across the displays) that were relatively automatic and effortless. Thus, these studies focused solely on the benefits due to the visual information gained (Eckstein, 2011; Epelboim & Suppes, 2001; Najemnik & Geisler, 2005; Renninger, Verghese, & Coughlan, 2007; Semizer & Michel, 2017). The present study examines strategies and the role of implicit costs, when more effortful and demanding actions were required. 
Present study
The present study examined the role of implicit costs connected to both motor effort and perceptual availability during the performance of a high-level and frequently encountered visual information-gathering task, namely, reading a graph and its accompanying text. 
The present study differs from most prior research on the role of implicit costs connected to visual information-gathering in that the prior work used tasks that operated over relatively short time scales (several seconds) and had singular, compact goals, such as finding one or more specified targets (e.g., Araujo et al., 2001; Ballard et al., 1995; Coëffé & O'Regan, 1987; Constantino & Daw, 2015; Hooge & Erkelens, 1998, 1999; Kibbe & Kowler, 2011; Navalpakkam et al., 2010; Wolfe, 2013; Wu & Kowler, 2013). By contrast, many of the information-gathering tasks people perform routinely require thinking and interpretation. Such tasks—and reading graphs is a good example—make demands on controlled processing and rest on accumulating and integrating information over relatively long time scales (Carpenter & Shah, 1998). With such higher-level task demands, there are two distinct and opposing ways of viewing how the implicit costs (e.g., time or mental effort) connected to planning or performing the actions could influence strategies. First, the need to develop a coherent interpretation of the visual display may dominate, and thus dictate what visual information is sought and when it is sought, regardless of the costs associated with the actions needed to acquire the information. Alternatively, and in accord with the arguments of Shenhav et al. (2017), the resources needed to interpret the displays may compete with the resources required to gather the information, thus encouraging a strategy in which more effortful or resource-consuming actions are avoided. 
The goal of the present study is to find out whether and how strategies of switching between viewing a graph and its accompanying text are influenced by changes to the types of actions required to switch between these two sources of information. Reading a graph is a demanding task requiring the accumulation of, and memory for, selected details to make decisions about the meaning of the material (Carpenter & Shah, 1998; Michal & Franconeri, 2017; Shah & Freedman, 2009). Graphs, such as those encountered in books, websites, or journal articles, are typically accompanied by some descriptive or explanatory text, where readers are expected to develop their own strategies of examining both the graph and text, including decisions about when to switch between them. Switches between modalities or attended features can be costly and aversive, even in relatively simple visual tasks (Kool et al., 2010; Monsell, 2003). Thus, any switches between graph and text may carry implicit costs even when the actions required to make the switch, such as gaze shifts between simultaneously visible regions, are fairly effortless and relatively automatic (Ross & Kowler, 2013; Wang & Pomplun, 2012). Costs attached to making the switches may be greater when the graph and text are available sequentially rather than visible simultaneously because additional motor actions (mouse clicks or page turns) are required with the sequential presentations, thus increasing the effort level (motor or mental). Sequential presentations also add to the implicit costs because of the greater demands on memory and on controlled, deliberate decisions when accessing a target region that is out of view (Funahashi, 2014; Wang, Cohen, & Voss, 2015). 
The present study varied the way in which switches between views of a graph and its accompanying text were carried out. Two kinds of factors were investigated: motor effort (gaze shift vs. button press) and perceptual availability (simultaneous vs. sequential presentations). Motor effort was studied because (1) prior work has shown that motor actions (mouse clicks) are more effortful than gaze shifts during a visual search task (e.g., Kibbe & Kowler, 2011) and (2) many common situations involving graphs and text require motor actions beyond a simple and relatively automatic gaze shift (mouse clicks or page turns, for example) to switch between them. Perceptual availability was examined because any differences between switches mediated by a gaze-shift vs. a button-press could be due either to the increased effort in planning or carrying out the action, or to the additional load on working memory or controlled processing required to access material that is currently out of view. Perceptual availability was manipulated by comparing performance in a condition where the graph and text were presented simultaneously, side by side, with a condition, termed eye-contingent, in which the graph and text were presented in the same locations, but visible only when a saccade was made into the relevant region. 
The task required reading bar graphs that depicted the value of two fictitious products along two different attributes to decide which product was preferred. This task was chosen because it required extensive inspection and interpretation, while minimizing the variability in performance due to prior specialized knowledge on the part of the viewer. The overall spatial layout of the graphs (four bars arranged in two groups) was kept the same in all trials. Other aspects of the content, to be described in Methods, were varied across trials so as to motivate a detailed inspection and to avoid stereotypical strategies. 
Decisions about when or how often to view the graph or the text were made by the viewer and, given that the decision reported at the end of the trial represented a preference, there were no formally correct or incorrect answers. There were no explicit incentives to favor one region (graph or text) over the other, and viewers could have formed a preference solely by inspecting either by itself. Strategic options could range from lengthy inspection of either the graph or the text with little or no switching, to a feature-by-feature inspection that entailed frequent switching. This research design, which gives substantial control of strategy to the viewer, is similar to that used in prior work on the role of costs (e.g., Araujo et al., 2001; Ballard et al., 1995; Coëffé & O'Regan, 1987; Kibbe & Kowler, 2011; Kool et al., 2010; Moher & Song, 2014) and allows opportunity for any influences of either motor effort or perceptual availability to become apparent through examination of the frequency and timing of the switches between graph and text. 
The main goal of the study was to determine whether and how the perceptual motor conditions influenced the choice of strategy. A second goal was to use the observed pattern of switches under the three perceptual motor conditions to infer aspects of the strategies used to integrate information across the graph and text. 
Methods
Eye movement recording
Movements of the right eye were recorded using the monocular EyeLink 1000 (SR Research, Osgoode, Canada), tower-mounted version, sampling at 1000 Hz. A chin rest was used to stabilize the head. Viewing was binocular. 
Subjects
There were 22 student subjects from Rutgers University, 11 tested in Instruction 1 and 11 in Instruction 2 (see Procedure section for definitions of instructions). All had normal vision and were naïve to the purpose of the experiment. An additional seven subjects were tested but data were not analyzed because: (1) at least 30% of the data were lost during the trials, mainly due to frequent blinks or periodic occlusion of the pupil by the eyelid (five subjects); (2) use of the buttons to switch between the graph and the text in the Button Press condition did not begin until late in the experimental session (two subjects), suggesting that they may not have understood the instructions. Testing was in accordance with the Declaration of Helsinki and approved by the Rutgers University Institutional Review Board. 
Stimulus display
Stimuli were displayed on a Dell U2413 LCD monitor (refresh rate 60 Hz; Dell, Round Rock, TX) viewed from a distance of 60 cm. Stimuli were displayed within a 1,280 × 1,024 pixel (28.2° × 22.5°) region of the screen. Stimuli were presented in a fully lighted room, allowing the boundaries of the display region to be seen at all times. Displays consisted of a bar graph and a paragraph of text. Examples are shown in Figures 1 and 2
Figure 1
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by attribute and values of the attributes are not in conflict. Text is redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Figure 1
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by attribute and values of the attributes are not in conflict. Text is redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Figure 2
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by product and values of the attributes are in conflict for the two products. Text is not redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Figure 2
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by product and values of the attributes are in conflict for the two products. Text is not redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Stimulus: Bar graphs
The bar graph stimuli were generated as 447 × 502 pixel (9.9° × 11.1°) images. The axes were contained within a 329 × 410 pixel (7.3° × 9.1°) rectangle. 
Graphs contained four colored bars on a white background. The bars compared the values of two fictitious common household products along two different attributes, with values of the two attributes shown on the left and right Y-axes, respectively. Lettering (legend and axis labels) was black. 
The stimulus configuration was varied in several ways to increase unpredictability and encourage a thorough inspection of the material. Bars were grouped in pairs, either according to the products or to the attributes. In the case of grouping by attribute (Figure 1) the labels on the X-axis under each pair indicated the name of the attribute, and the color of the bars indicated the product. In the case of grouping by product (Figure 2), the labels on the X-axis under each pair indicated the name of the product, and the color of the bars indicated the attribute. In addition, the relative merits of the two products on each attribute either were (Figure 2) or were not (Figure 1) in conflict, where a conflict meant that one item was superior to the other on only one of the two attributes. 
Twenty-four different bar graphs were generated, each with a different product pair. Then, four versions of each of these 24 graphs were generated, according to whether (1) the values of the attributes were either in conflict or not in conflict, and (2) the bars were grouped by product or by attribute. 
Text
Each graph was accompanied by a paragraph of text, 264 to 387 characters in length (including spaces). Text was black 18-point monospaced Courier New font (width = 13.6 pixels/character, 6 characters/°) on a white background. The text was displayed within a 555 × 760 pixel region (12.3° × 16.8°). Each of the four versions of graph (see Graph Stimuli) could be accompanied by one of two types of text, redundant (Figure 1) or nonredundant (Figure 2). Variation in the characteristics of the text, like the variations in the configuration of the graph, was implemented to discourage the use of the same stereotypical strategy of viewing graph and the text on each trial. Redundant text restated the information depicted in the graph. Nonredundant text provided information that differed in some way from the information depicted in the graph. This information was either irrelevant to the relative merits of the items, or added details that favored one of the products, or stated that the information depicted in the graph was outdated or contained an error. 
Note that the study was focused on the inspection strategies, and not on understanding the preferences, or whether the choice was “correct.” In at least half of the cases (conflict between attributes), there was no obvious “correct” choice, and in the remaining cases viewers could evaluate the attributes in any way they chose. 
Perceptual motor conditions
Three perceptual motor conditions were tested, shown in Figure 3
Figure 3
 
Perceptual motor conditions. Left: Simultaneous: Graph and text appear side by side. Center: Button Press: Graph and text are centered, with the sequential appearance of each triggered via a button press. Right: Eye-Contingent: Graph and text appear side by side, with the sequential appearance of each triggered via a saccade into the corresponding area of the screen.
Figure 3
 
Perceptual motor conditions. Left: Simultaneous: Graph and text appear side by side. Center: Button Press: Graph and text are centered, with the sequential appearance of each triggered via a button press. Right: Eye-Contingent: Graph and text appear side by side, with the sequential appearance of each triggered via a saccade into the corresponding area of the screen.
  1.  
    Simultaneous: Graph and text were displayed side by side. The regions containing the graph and the text were separated by a blank region 16 pixels (0.35°) wide (see Figures 1 and 2). The graph was displayed either on the right or left side, randomly and independently chosen on each trial. Informal testing verified that the critical details of the text or graph (such as the words of the text, legends, or axis labels) could not be identified when the opposite region was fixated, and that saccades would be needed to inspect each region to discern the details.
  2.  
    Button Press: Graph and text were displayed sequentially, each located in the center of the screen. Subjects pressed a trigger button on a gamepad to display the graph or the text.
  3.  
    Eye-Contingent: Graph and text were displayed sequentially, as in the Button Press condition, and appeared either on the right or on the left side of the display, randomly chosen for each trial, in the same locations as in the Simultaneous condition. The appearance of the graph or text was triggered by online detection of the offset of a saccade into the right or the left side of the screen. The empty region contained no visual details except for the boundaries of the white display region itself. The delay between the offset of the triggering saccade and onset of the display, averaged across 24 representative trials, was 68 ms (SD = 19) for all except the very first appearance of text, for which the mean delay was 121 ms (SD = 13).
Procedure
Each subject was tested in a single 24-trial experimental session. Before testing began subjects were told they would be viewing a series of graphs accompanied by passages of text about two products. At the end of each trial they were asked to indicate which product they preferred. Two groups of subjects (11/group) were tested. The first group was told they should read the graph and text to determine their preference (Instruction 1). The second group was given instructions that did not contain the word “read” to avoid implying that the text was more important than the graph. They were told only to indicate their preferred product based on the display (Instruction 2). Subjects were told they could end the trial by pressing a button on the gamepad when they were ready to make the decision. 
Before testing began subjects were presented with three familiarization trials, one for each of the perceptual motor conditions, in order to illustrate the perceptual motor conditions and give the subjects ample opportunity to practice switching the display between graph and text using either a button press (Button Press condition) or a saccade (Eye-Contingent condition). A diagram indicating which button corresponded to the graph and which to the text was available throughout the experimental session. 
The calibration routine built into the EyeLink software was run before the start of each experimental session and again midway through. Before each trial, the number of the trial as well as a label indicating the perceptual motor condition was displayed. Subjects started the trial with a button press when ready. Then, as an additional check on the calibration, five crosses were presented for 5 s, one in the center and one in each corner of the display. Subjects were told to fixate the center cross, then look to each of the other four crosses in sequence, then back to the center cross. The crosses then disappeared, replaced by the critical display. 
In the Button Press condition, the button used to start the trial determined whether the graph or text appeared first. In the Eye-Contingent condition, the screen was blank until subjects fixated either side, at which point the graph or the text appeared. Thus, in the Eye-Contingent condition, subjects did not know which side corresponded to each until after the first saccade. 
Subjects were instructed to press a button to end the trial when they were ready to make the decision about their preferred item. The trial automatically ended after 2 minutes if the subjects did not choose to end it themselves (only three of the 528 total trials tested lasted the full 2 minutes). The fixation of the five crosses was repeated. Then, the subject indicated by button presses: (1) which item they preferred, (2) how confident they were in their choice on a scale of 1 to 4 (with 1 being least confident and 4 being most confident), and (3) which of the two attributes was more influential in their choice. These questions were asked to motivate the inspection of the graph and text. Analysis of whether fixated locations predicted the decisions were outside the scope of the present study. 
Design
The perceptual motor conditions were assigned to each trial randomly using an algorithm that employed the following constraints: (1) Each subject was tested on eight trials for each of the three perceptual motor conditions. The order of testing trials with the different perceptual motor conditions was random. (2) The pairing between a given graph and a given perceptual motor condition was different for each subject. The pairing was done so that across subjects each of the 24 graphs was paired with a given perceptual motor condition at least once for Instruction 1 and at least once for Instruction 2. (3) Text conditions (redundant vs. not redundant, see Text Stimuli) were tested in blocks of six trials each, with the first block chosen at random. (4) Graph type (conflict vs. no conflict; grouping by product vs. grouping by attribute) and the side of the screen containing the graph were independently chosen at random on each trial. 
Analysis
All analyses were carried out in Matlab (MathWorks, Natick, MA). The beginning and ending positions of saccades were detected offline by means of a custom-written algorithm employing a velocity criterion to find saccade onset and offset. The value of the velocity criterion (22°/s) was verified empirically for individual observers by examining a large sample of analog recordings of eye positions. Portions of data containing blinks or episodes where the tracker signal was lost were eliminated. Trials in which lock was lost more than half the time were eliminated. Of the 528 trials tested (176/perceptual motor condition), four were eliminated in the Simultaneous condition, 11 in the Button press condition, and four in the Eye-contingent condition. 
The location of each fixation in the display was determined from the average position of the line of sight at the offset of saccade n-1 and the onset of saccade n. Each fixation pause was classified as being in the graph, text, or neither. 
Consecutive fixations of the same region were cumulated into visits. A visit was defined as a block of time inspecting either graph or text, where the block of time was composed of the accumulation of successive fixations in the same area, including the time during any gaze shifts, as well as any intervening blinks. 
Examination of individual trials showed that some contained brief (<1 s) visits at the beginning of the trial, followed by longer inspections (see Figure 6 for examples). Given that long visits occurred following these very brief initial visits, it was likely that the region viewed during these brief initial visits was not used for the purpose of gathering information. A single brief (<1 s) initial visit occurred in 45% of trials in the Simultaneous condition, 1% of trials in the Button Press condition, and 5% of trials in the Eye-Contingent condition. A total of 12% of trials in the Simultaneous condition, and <1% in the Eye-Contingent condition, contained more than one initial brief visit. To avoid incorporating such brief initial views, and thereby inflating the number of visits to graph or text in the Simultaneous condition, the analysis of results for a given trial did not include these initial brief visits. 
Fixation pauses and visits were further categorized as falling into one of seven areas of interest (AOI's) shown in Figure 4. These AOI's were selected because each contained a different type of information relevant to interpreting the graph. Boundaries of the AOI's were defined manually (see Figure 4). Fixations landing in the narrow region between the graph and text, above the graph (such as near the title), or below the graph were classified as “other.” 
Figure 4
 
Location of the boundaries of the seven AOI's in the display.
Figure 4
 
Location of the boundaries of the seven AOI's in the display.
Results
Effect of perceptual-motor condition on visits to graph and text
Performance was compared across three perceptual-motor conditions: (1) Simultaneous: Graph and text were displayed side by side, requiring a saccadic eye movement to switch between them; (2) Button Press: Graph and text were displayed sequentially at screen center, requiring the press of a button to switch between them. This condition required more motor effort (a button press instead of a saccade), and reduced the perceptual availability of each portion of the display, since only one portion, graph or text, was presented at a time; (3) Eye-Contingent: Graph and text appeared sequentially in the same locations as in the Simultaneous condition, with the appearance of each triggered by a saccade into the corresponding region on the screen. The Eye-Contingent condition thus reduced the perceptual availability of the graph and the text due to the sequential aspect of the presentation, as in the Button Press condition, without adding motor effort beyond the saccade, as in the Simultaneous condition. 
The main result was that the pattern of visits to the graph and text depended on the perceptual motor condition. There were about twice as many visits/trial to graph and text in the Simultaneous condition than in either the Button Press or the Eye-Contingent conditions. This result can be seen in Figure 5 (left), which compares the mean number of visits/trial to the graph and to the text (means of subject means) for the three perceptual motor conditions for both instruction types (see Analysis section for the definition of a visit and the rule for designating the first visit of a trial). Analysis of variance (3 perceptual-motor conditions × 2 instruction types) showed a significant effect of perceptual-motor condition, F(2, 40) = 24.01, p = 10−7, and no significant effect of the instruction type, (Procedure section), F(1, 20) = 0.52, p = 0.48. The pattern of results was the same when results were first averaged within each of the 24 different graphs (Graph Stimuli section) instead of within individual subjects (Supplementary Figure S1). 
Figure 5
 
Left: Mean number of visits/trial to graph and text (±SE). Center: Mean duration (in seconds) of visits to graph and text (±SE). Right: Mean visits/second (±SE). Data are shown for the three perceptual motor conditions and the two instruction types. Means are based on means of subject means, 11 subjects for each instruction. See Supplementary Tables S1 and S2 for individual subject data, and Supplementary Figure S2 for distributions of the number of visits/trial.
Figure 5
 
Left: Mean number of visits/trial to graph and text (±SE). Center: Mean duration (in seconds) of visits to graph and text (±SE). Right: Mean visits/second (±SE). Data are shown for the three perceptual motor conditions and the two instruction types. Means are based on means of subject means, 11 subjects for each instruction. See Supplementary Tables S1 and S2 for individual subject data, and Supplementary Figure S2 for distributions of the number of visits/trial.
Figure 6
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 1 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds to the duration of the visit (accumulation of fixations) to the graph or text. Arrows on the Simultaneous timelines show examples of trials with brief glances to graph or text at the beginning of the trial before a longer visit to the other region (see Analysis section). Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Figure 6
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 1 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds to the duration of the visit (accumulation of fixations) to the graph or text. Arrows on the Simultaneous timelines show examples of trials with brief glances to graph or text at the beginning of the trial before a longer visit to the other region (see Analysis section). Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Figures 5 and Supplementary Figure S1 also show that the average number of visits/trial in the Eye-Contingent condition was slightly, but reliably, larger than in the Button Press condition (paired t test: Eye-Contingent vs. Button Press; t(21) = 3.4, p = 0.0029). The average number of visits/trial in the Simultaneous condition was about twice that of the sequential conditions. This result suggests that while motor effort (gaze shift vs. button-press) was influential, the simultaneous versus sequential availability of the graph and the text played a larger role in determining the occurrence of switches. 
Average trial duration did not differ across the three perceptual motor conditions (Figure 5, Supplementary Figure S1). Analysis of variance (3 perceptual motor conditions × 2 instruction types) showed no significant effect of perceptual motor condition, F(2, 40) = 1.06, p = 0.36, nor instruction type, F(1, 20) = 0.65, p = 0.43, on trial duration. Figure 5 and Supplementary Figure S1 also show that time was apportioned about equally between graph and text for each of the three perceptual motor conditions. Finally, the confidence ratings attached to the choices of the preferred product, a measure that might have been sensitive to the value of the information acquired, were almost the same for the three perceptual motor conditions (mean confidence rating for the Simultaneous condition was 3.39 (SD 0.49), Button Press was 3.42 (SD 0.38) and Eye-Contingent was 3.47 (SD 0.40). 
The effect of perceptual motor condition on performance can be summarized by the mean rate of visits (number of visits per trial/trial duration), which was about two times greater in the Simultaneous condition than in the other two perceptual motor conditions (Figure 5, Supplementary Figure S1). Effects of perceptual motor condition were once again significant, F(2, 40) = 62.56, p = 10−13, and effects of instruction type were not, F(1, 20) = 1.33, p = 0.26. 
In summary, the perceptual motor conditions affected how time was apportioned between the graph and the text, with the Simultaneous condition characterized by a higher rate of switching between the graph and the text than either the Button Press or the Eye-Contingent conditions. The perceptual motor condition did not affect the total amount of time devoted to inspecting graph and text, only how the time was apportioned into separate visits. The sequential availability of graph and text in both the Button Press and Eye-Contingent conditions played a greater role than motor effort in discouraging switching between the regions. 
Effect of the variations in the configuration of graph and text
Variations were introduced into the configuration of the stimuli to discourage the development of stereotypical strategies and encourage extensive examination of the display (see Graph Stimuli and Text sections). None of these manipulations affected the rate of visits. Specifically: (1) whether the bars were grouped by Item or by Attribute, F(1, 20) = 0.77, p = 0.39; Instruction type: F(1, 20) = 1.17, p = 0.29; (2) whether the relative values of the two items on each attribute were consistent or in conflict, F(1, 20) = 0.89, p = 0.36; Instruction type: F(1, 20) = 0.95, p = 0.34; or (3) whether the text was redundant with the graph or introduced new or different information, F(1, 20) = 3.74, p = 0.068; Instruction type: F(1, 20) = 1.47, p = 0.24. In the case of redundancy, there were 0.13 visits/s (SD 0.037), N = 22. For nonredundant text there were 0.14 visits/s (SD 0.035), N = 22. 
Timelines
Timelines were constructed to visualize the sequence of visits to graph and text for each perceptual motor condition. The timelines show the sequence and duration of the visits to the graph and the text for each trial, ordered from shortest to longest, for Instruction 1 (Figure 6) and Instruction 2 (Figure 7). As was the case in Figure 5, a “visit” was composed of the accumulation of successive fixations in the same area, graph, or text, including the gaze shift time, as well as any intervening blinks. Any visits to locations other than the graph or the text, including fixations in the blank region between the graph and the text, are shown in red. Blank regions of the timelines indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s. All visits, including initial visits <1 s (see Analysis section), are shown. 
Figure 7
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 2 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds the duration of the visit (accumulation of fixations) to the graph or text. Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Figure 7
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 2 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds the duration of the visit (accumulation of fixations) to the graph or text. Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Inspection of the timelines suggests two trends, which will be examined in more detail in the Visit durations section. First, most trials started with a visit of several seconds to either the graph or the text, after which gaze switched over for a visit of several seconds to the other region. Second, these initial long visits were often followed by shorter duration visits to the graph or the text. These shorter visits occurred more frequently in the Simultaneous condition. 
Visit durations
Figures 8 and 9 show distributions of the durations of the first three visits to graph or text for trials in which the first visit was to the graph (Figure 8) or to the text (Figure 9). As noted earlier (Analysis section), analyses began with the first visit to graph or text that was longer than 1 s. 
Figure 8
 
Histograms showing the durations of the first three visits for all trials in which the graph was visited first. The sequence was graph (top), text (middle), and graph again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
Figure 8
 
Histograms showing the durations of the first three visits for all trials in which the graph was visited first. The sequence was graph (top), text (middle), and graph again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
Figure 9
 
Histograms showing the durations of the first three visits for all trials in which the text was visited first. The sequence was text (top), graph (middle) and text again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
Figure 9
 
Histograms showing the durations of the first three visits for all trials in which the text was visited first. The sequence was text (top), graph (middle) and text again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
The first visit to either the graph or the text typically lasted several seconds, regardless of which region was visited first. These durations were long enough to allow extensive examination of the graph or the text. These findings suggest that the preferred strategy during the initial visit was to attempt to extract meaning from large portions of text or from entire graph, rather than a strategy of inspecting the graph and text jointly, feature by feature. There was also a suggestion of a small effect of order in that the initial visits to the text were shorter when the graph was visited first (Figure 8) than when the text was visited first (Figure 9) in all three perceptual motor conditions, suggesting that reading the graph may have either helped or supplanted in part the reading of the text. 
Figures 8 and 9 show how the perceptual motor condition affected the duration of the visits. In all cases the average durations of the visits were shortest in the Simultaneous condition. When the graph was visited first (Figure 8), there was a significant effect of perceptual-motor condition for each of the first three visits, shown in each row of the figure, Visit 1: F(2) = 15.78, p = 10−7; Visit 2: F(2) = 4.84, p = 0.0089; Visit 3: F(2) = 6.47, p = 0.020. When the text was visited first (Figure 9), visits were also shortest in the Simultaneous condition; however, the effect of perceptual motor condition was significant only for the second visit, Visit 1:F(2) = 2.25, p = 0.107; Visit 2: F(2) = 8.659, p = 0.00023; Visit 3: F(2) = 5.17, p = 0.067. The distributions also show the trend that was apparent in the timelines (Figures 6 and 7), namely, after the first visit to the graph (Figure 8) or to the text (Figure 9) the Simultaneous condition was characterized by frequent follow-up visits of short duration (<3 s). 
Which region was visited first?
In the Simultaneous condition, the text was usually visited first (Table 1). In the other two conditions, preferences to visit graph or text first were about equal. Thus, the simultaneous availability of both regions disclosed a preference for the text. The tendency to visit the text first in the Simultaneous condition was significant only for subjects in Instruction 1 (t(10) = 7.87, p = 10−5; Instruction 2: t(10) = 1.71, p = 0.12), where the word “read” was included in the instructions. 
Table 1
 
Means of subject means of proportion of trials in which the text was visited first (for visits >1 s).
Table 1
 
Means of subject means of proportion of trials in which the text was visited first (for visits >1 s).
In the Eye-Contingent condition the side of the display containing the graph or the text was not known until it was fixated. Thus, the behavior in the Eye-Contingent condition allowed an examination of the extent to which an extra saccade to view a preferred region would be made if the initial fixation landed on a nonpreferred region. To address this question, the proportion of trials viewers stayed with or switched from the region fixated first was examined for the Eye-Contingent condition. This analysis included any initial brief (<1 s) glances to either graph or text (Analysis section). 
Subjects rarely decided to switch from the region initially fixated in the Eye-Contingent condition. Switches occurred on 9% of the trials where graph was fixated first, and 1% of the trials in which the text was fixated first. This willingness to leave the choice of first location to chance in the Eye-Contingent condition is another indication of the influence of perceptual availability on the viewing strategy. 
Strategy of comparing graph and text
What was the underlying strategy behind the transitions between graph and text? One possibility is that the strategy was dominated by a partitioning of the material, for example, reading only part of the graph or text in the initial visit, and then switching to the other region. Alternatively, it is possible that the initial visit was devoted to a more exhaustive inspection and follow-up visits were devoted to re-examining of previously seen regions of either the graph or the text. 
To distinguish these possibilities, the locations of visits within graph and within text were analyzed. A “re-examination” within the graph was defined as a revisit of one of the six AOI's (see Figure 4 for AOI's). A re-examination within the text was defined as re-reading of all or part of a line of text. 
Figure 10 shows the average time spent viewing new areas (blue) and re-examining old areas (red) of the graph (top) or the text (bottom). Results are shown separately for each of the first three visits, and pooled over the entire trial. Each section of a bar represents the mean over subjects of the average time spent per trial viewing graph or text. The number of subjects on which each mean is based is shown above each bar. (Note that these numbers differ because some subjects had no trials with more than one visit to graph or text, typically, in the Button Press or Eye-Contingent conditions). Trials with more than three visits are not shown because there were not enough of such trials in the Button Press or Eye-Contingent conditions to allow meaningful comparisons across the perceptual motor conditions. 
Figure 10
 
Mean time (± SE) spent visiting new AOI's of the graph or new lines of text (blue) and re-examining previously visited AOI's or lines of text (red) for the first three visits and over the whole trial (All) for visits to the graph (top) and text (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Numbers above each bar represent the number of subjects that contributed to the means. A subject was not included if they did not have any second or third visits to graph or text, respectively, in that condition.
Figure 10
 
Mean time (± SE) spent visiting new AOI's of the graph or new lines of text (blue) and re-examining previously visited AOI's or lines of text (red) for the first three visits and over the whole trial (All) for visits to the graph (top) and text (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Numbers above each bar represent the number of subjects that contributed to the means. A subject was not included if they did not have any second or third visits to graph or text, respectively, in that condition.
There were two trends that were found in all three perceptual-motor conditions. First, a substantial portion of the time was spent re-examining previously seen material. The total time spent re-examining portions of the graph or the text, cumulated over all visits (bars labeled “All” in Figure 10) was about the same for all perceptual-motor conditions (means were 80% for Simultaneous, 77% for Button Press, and 78% for Eye-Contingent conditions). Analysis of variance showed no significant differences among the three perceptual motor conditions for the percent of time spend re-examining the graphs, F(2, 40) = 0.191, p = 0.83, or the Text, F(2, 40) = 0.1713, p = 0.84. 
The second major trend evident in Figure 10, again observed for each of the three perceptual motor conditions, was that the examination of new material occurred almost exclusively during the initial visit to graph or to text. New material was almost never examined after the initial visit. 
The time devoted to new text during the initial visit was about the same across all three perceptual-motor conditions, F(2, 40) = 1.51, p = 0.234. The time devoted to new portions of the graph differed slightly, but significantly, across the three perceptual motor conditions, F(2, 40) = 13.86, p = 10−5, with the time shorter in the Simultaneous condition than in the other two. 
The preference to examine new material during the initial visit, rather than during subsequent visits, shows that switches between graph and text, even in the Simultaneous condition, were avoided until completion of at least an initial inspection of one of the regions. 
Differences among the perceptual motor conditions were most apparent in the portions of the first visits devoted to re-examining material. The time devoted to re-examination differed across conditions both for the first visits to the text, F(2, 40) = 6.087, p = 0.0049), and first visits to the graph, F(2, 40) = 13.95, 10−5. As can be seen in Figure 10, there was less time devoted to re-examinations in the Simultaneous condition than in either the Button Press or the Eye-Contingent conditions. 
This result, along with the fact that the total time spent re-examining old material was the same across conditions (see bars labeled “All” in Figure 10), show that the perceptual motor conditions affected the strategy of re-examination. Specifically, the same total time devoted to re-examination was spread across more frequent and shorter visits in the Simultaneous condition, and compacted into less frequent and longer visits in the Eye-Contingent and Button Press conditions. 
Strategies of viewing the graph
To determine whether the perceptual motor condition had any major influences on the viewing of the graph, matrices showing the number of transitions between Areas of Interest (AOI's) (Figure 4) were constructed for all three perceptual motor conditions (Figure 11). The most frequent transitions in all three conditions occurred between the labels of the axes and the bars, and between the bars and the legend, reflecting a strategy of frequent re-fixations of the referents (Carpenter & Shah, 1998). 
Figure 11
 
Matrices depicting the frequency of transitions between different areas of interest (AOI) within the graph or the text for all trials for each of the three perceptual motor conditions. The rows indicate the AOI of the previous visit and the columns indicate the AOI of the current visit. The color of each cell depicts the number of transitions from the AOI of the previous visit to the current visit. L = legend; YL = Y-axis label of left pair of bars; YR = Y-axis label of right pair of bars; BL = left pair of bars; BR = right pair of bars; X = X-axis label; T = title.
Figure 11
 
Matrices depicting the frequency of transitions between different areas of interest (AOI) within the graph or the text for all trials for each of the three perceptual motor conditions. The rows indicate the AOI of the previous visit and the columns indicate the AOI of the current visit. The color of each cell depicts the number of transitions from the AOI of the previous visit to the current visit. L = legend; YL = Y-axis label of left pair of bars; YR = Y-axis label of right pair of bars; BL = left pair of bars; BR = right pair of bars; X = X-axis label; T = title.
Discussion
One approach to gathering information from a graph and its accompanying text would be to sample information solely on the basis of its value for interpreting the depicted material without taking into account the characteristics of the actions needed to gain access to the material. In cases where the required actions are fast, simple, or relatively automatic, as may be the case for most shifts of gaze across a visual display, the decision to base strategies solely on the information content represents a rational choice, one supported by research showing the role of the value of the information gained in determining the locus of gaze when searching visual displays (Eckstein, 2011; Najemnik & Geisler, 2005; Renninger et al., 2007; Semizer & Michel, 2017). 
The present study showed that strategies for gathering information from a graph and its accompanying text were not based solely on the information content, but were instead influenced by characteristics of the actions needed to gain access to the material. Switches between the graph and the text were less frequent when carried out by a button press rather than by a gaze shift. Switches were also less frequent when the graph and text were presented sequentially, rather than simultaneously, with the contents of each region revealed only after gaze landed within the region (termed the Eye-Contingent condition). In addition, switches were found mainly during the portion (∼80%) of trials devoted to re-reading material. Switches, even in the case of simultaneous availability, were rare during the initial readings of graph or text. 
The perceptual motor conditions did not affect either the total amount of time taken to complete the task, the overall proportion of time devoted to the graph and the text, or the confidence of the judgments. Thus, the perceptual motor condition did not affect the choices about how much information to sample from the graph or the text. Rather, the perceptual motor condition affected choices about when to take these samples. 
The finding that motor effort and perceptual availability affected the rate of switching between the graph and the text is consistent with prior studies using simpler visual tasks that showed how adding to the implicit costs of planning or carrying out the required actions affected the strategies used to gather visual information (Araujo et al., 2001; Ballard et al., 1995; Hooge & Erkelens, 1998, 1999; Kibbe & Kowler, 2011). The present findings extend these results to a task that requires considerable thinking and interpretation. Thus, the greater cognitive load attached to reading a graph and text, by itself, did not preclude a role for motor effort or perceptual availability. The greater cognitive load attached to interpreting the graph and text may have contributed to any inherent aversion to effortful switches because of a greater level of competition for access to limited processing resources (Shenhav et al., 2017). 
Switches during the initial view and during re-examination
Carpenter and Shah (1998), in their study of eye movements during reading of graphs, distinguished two main phases of reading graphs: an initial inspection to obtain an overview of the graph, followed by repetitive scanning of key features (especially the referents) to develop an interpretation of the depicted material. 
We found that the rate of switches was different for these two phases. The first phase, obtaining the initial overview of either the graph or the text, contained almost no switches, even in the Simultaneous condition. Analysis of the durations of visits to new material and re-examination of old material (Figure 10) showed that virtually all the viewing of new portions of the graph or text occurred uninterrupted within the first visit to each region. This finding is consistent with the findings in studies of perceptual switch costs, which reported increases in reaction time when the relevant perceptual features of the visual discrimination task changed from one block of trials to the next (Monsell, 2003). These increases in reaction time were attributed to the need for extra processing steps to change the mental set or task set to different features or attributes. Switches between graph and text may be accompanied by comparable difficulties due to changes in task set, leading to preferences to avoid switches (similar to Kool et al., 2010), at least during the initial readings of each region. 
Switches between graph and text began to occur during the re-examination of previously viewed material. The re-examination (Figure 11) consisted of repeated fixations of details of the graph (Carpenter & Shah, 1998), as well as re-reading of portions of the text (Schnitzer & Kowler, 2006). Switches may have become more frequent during re-examination because re-examination may demand different types of mental resources than the initial inspection. For example, during re-examination strategies may be driven by the testing of hypotheses about content, rather than by an attempt to develop an initial understanding of what was depicted in the graph. 
Role of perceptual availability
The sequential presentation of graph and text in both the Button Press and Eye-Contingent conditions led to a lower rate of switches between the regions than was found for the simultaneous presentations. We can view these findings as showing that simultaneous availability promotes switching, or, equivalently, that sequential availability discourages switching. We consider each viewpoint as follows. 
In the Simultaneous condition, the continuous presence of each region in the visual field may have encouraged or facilitated switching in a relatively automatic manner. For example, the visible eccentric region could have served as a constant cue or reminder that the region was available and might contain some potentially useful information. This view is similar to that proposed in Wang and Pomplun (2012) and Ross and Kowler (2012), who found that gaze was directed to displayed text even when viewing the text was not necessary for completing the task. In the case of the graph and the text in the current study, the eccentric region may receive some level of spatial attention, given its task relevance, which could increase the probability of a saccade (Gersch, Kowler, Schnitzer, & Dosher, 2009; Zhao, Gersch, Schnitzer, Dosher, & Kowler, 2012). At a neural level, continuous activation of a visible region in neural areas involved in spatial attention and saccades, such as FEF, LIP, or SC (Awh, Armstrong, & Moore, 2006; Basso & May, 2017; Gottlieb, 2007; Schall, 2004) could contribute to eliciting spontaneous shifts of attention or spontaneous shifts of gaze to the opposite region with minimal reliance on overt or effortful decisions. The relatively effortless shifts of attention or shifts of gaze to visible regions could also be encouraged by mechanisms that increase the probability of saccades to visible regions associated with higher levels of explicit rewards or with information that reduces uncertainty (Gottlieb, Hayhoe, Hikosaka, & Rangel, 2014). 
On the other side of the coin, the absence of an immediate visual representation of the graph or text during either of the sequential conditions, Button Press or Eye-Contingent, would mean that the conditions that facilitated effortless or automatic shifts of gaze were absent. Thus, additional processing steps would likely be involved, such as the retrieval of information about the type of content (text or graph) from memory, an expectation or prediction about what might be encountered following the switch, or an overt, controlled decision to make the switch. These types of operations fall under the broad category of executive functions that require management and monitoring by areas such as prefrontal cortex (PFC), anterior cingulate cortex (ACC), or hippocampus (Buschman & Miller, 2014; Funahashi, 2014; Miller & Cohen, 2001; Shenhav, Botvinick, & Cohen, 2013; Voss, Bridge, Cohen, & Walker, 2017). Wang et al. (2015), for example, proposed that links between PFC and hippocampus are involved in managing actions to access material not currently in view. The management of the additional processing steps required to direct an action to a region that is not immediately in view may have been sufficiently demanding of resources to discourage frequent switches between graph and text. 
Summary, conclusions, and implications
Reading a graph and text to arrive at a coherent interpretation of the content is a resource-intensive task that is encountered in many real-world settings. The present study investigated the roles of motor effort and perceptual availability in the integration of information between a graph and accompanying text. We found that both motor effort and perceptual availability influenced the rate of switches between the graph and the text. Thus, strategies of gathering visual information in a task with a strong requirement for thinking and interpretation were not based solely on the information content. The results suggest that factors connected to planning or executing the switches, such as effort level, dependence on working memory, or reliance on controlled rather than automatic processing, were taken into account when developing strategies of gathering information needed to interpret the graph and text. 
These results raise a very basic question. People often plan actions to regions that are not perceptually available. We turn pages of books, click mice, or press keys to access new web pages, or we turn our heads to view what is behind us. In everyday life, many of these behaviors have to be carried out despite any extra processing steps that proved to be an obstacle in the present task. What factors increase our willingness to engage in effortful actions? Two types of factors are likely to be relevant. 
The first is the importance of the visual information. For example, turning the head to glance at a side-view mirror while changing driving lanes is clearly a case in which the information is vital. Another example would be reading graphs when the necessary referents, such as axis labels or legends, which are fixated frequently (Figure 11; also, Carpenter & Shah, 1998), are kept out of view unless revealed by a motor action. The importance of such task-critical regions might induce viewers to undertake the effortful actions to gain access to the material. Such decisions may be part of a mental cost-benefit analysis (Kool et al., 2010; Leider & Griffiths, 2017) that people may carry out continually, even on short time scales, during performance of visuomotor tasks. Whether the increased importance or the value of the information affects the decision criteria for trading-off costs and benefits, or changes the perceived or actual level of mental effort, is an interesting question for future research. 
The second factor that would promote a greater tolerance for effortful actions is the development of well-learned visuomotor routines (e.g., Land & Hayhoe, 2001; Sailer, Flanagan, & Johansson, 2005). When applied to cases in which actions must be performed to reveal regions that are currently out of view, the effort level accompanying the actions may be reduced due to the use of encapsulated or habitual patterns of movements. 
Finding that even actions as innocuous as mouse-clicks or gaze shifts can affect the strategies of viewing graph or text suggests that people are continually evaluating the resource load when determining whether or when to seek out new visual information. These analyses may influence performance of a host of visuomotor tasks and be a major factor in determining natural behaviors. 
Acknowledgments
Research was supported by NSF DGE 0549115 and NSF DUE 1062735. We thank Cordelia Aitkin, Elsie Lee, Jie Wang, Melchi Michel, Pernille Hemmer, and Morgan McCabe for their useful suggestions and comments. 
Commercial relationships: none. 
Corresponding author: Eileen Kowler. 
Address: Department of Psychology, Rutgers University, Piscataway, NJ, USA. 
References
Araujo, C., Kowler, E., & Pavel, M. (2001). Eye movements during visual search: The costs of choosing the optimal path. Vision Research, 41, 3613–3625.
Awh, E., Armstrong, K. M., & Moore, T. (2006). Visual and oculomotor selection: links, causes and implications for spatial attention. Trends in Cognitive Science, 10 (3), 124–130.
Ballard, D. H., Hayhoe, M. M., & Pelz, J. B. (1995). Memory representations in natural tasks. Journal of Cognitive Neuroscience, 7 (1), 66–80.
Basso, M. A., & May, P. J. (2017). Circuits for action and cognition: A view from the superior colliculus. Annual Review of Vision Science, 3, 197–226.
Buschman, T. J., & Miller, E. K. (2014). Goal-direction and topic-down control. Philosophical Transactions of the Royal Society B, 369 (20130471), 1–9.
Carpenter, P. A., & Shah, P. (1998). A model of the perceptual and conceptual processes in graph comprehension. Journal of Experimental Psychology: Applied, 4 (2), 75–100.
Coëffé, C., & O'Regan, J. K. (1987). Reducing the influence of non-target stimuli on saccade accuracy: Predictability and latency effects. Vision Research, 27 (2), 227–240.
Cohen, E. H., Schnitzer, B. S., Gersch, T. M., Singh, M., & Kowler, E. (2007). The relationship between spatial pooling and attention in saccadic and perceptual tasks. Vision Research, 47 (14), 1907–1923.
Constantino, S. M., & Daw, N. D. (2015). Learning the opportunity cost of time in a patch-foraging task. Cognitive, Affective & Behavioral Neuroscience, 15 (4), 837–853.
Dean, M., Wu, S. W., & Maloney, L. T. (2007). Trading off speed and accuracy in rapid, goal-directed movements. Journal of Vision, 7 (5): 10, 1–12, https://doi.org/10.1167/7.5.10. [PubMed] [Article]
Eckstein, M. P. (2011). Visual search: A retrospective. Journal of Vision, 11 (5): 14, 1–36, https://doi.org/10.1167/11.5.14. [PubMed] [Article]
Epelboim, J., & Suppes, P. (2001). A model of eye movements and visual working memory during problem solving in geometry. Vision Research, 41 (12), 1561–1574.
Findlay, J. M. (1982). Global visual processing for saccadic eye movements. Vision Research, 22 (8), 1033–1045.
Funahashi, S. (2014). Saccade-related activity in prefrontal cortex: Its role in eye movement control and cognitive function. Frontiers in Integrative Neuroscience, 8, 54, 1–13.
Gersch, T. M., Kowler, E., Schnitzer, B. S., & Dosher, B. A. (2009). Attention during sequences of saccades along marked and memorized paths. Vision Research, 49 (10), 1256–1266.
Gottlieb, J. (2007). From thought to action: the parietal cortex as a bridge between perception, action, and cognition. Neuron, 53 (1), 9–16.
Gottlieb, J., Hayhoe, M., Hikosaka, O., & Rangel, A. (2014). Attention, reward and information seeking. Journal of Neuroscience, 34 (46), 15497–15504.
Hayhoe, M. M. (2017). Vision and action. Annual Reviews of Vision Science, 3, 4.1–4.25.
He, P. Y., & Kowler, E. (1989). The role of location probability in the programming of saccades: Implications for “center-of-gravity” tendencies. Vision Research, 29 (9): 1165–1181.
Hooge, I. T., & Erkelens, C. J. (1998). Adjustment of fixation duration in visual search. Vision Research, 38 (9): 1295–302.
Hooge, I. T., & Erkelens, C. J. (1999). Peripheral vision and oculomotor control during visual search. Vision Research, 39 (8), 1567–1575.
Kibbe, M. M., & Kowler, E. (2011). Visual search for category sets: Tradeoffs between exploration and memory. Journal of Vision, 11 (3): 14, 1–21, https://doi.org/10.1167/11.3.14. [PubMed] [Article]
Koehler, K., & Eckstein, M. P. (2017). Beyond scene gist: Objects guide search more than scene background. Journal of Experimental Psychology: Human Perception and Performance, 43 (6), 1177–1193.
Kool, W., McGuire, J. T., Rosen, Z. B., & Botvinick, M. M. (2010). Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology: General, 139 (4), 665–682.
Land, M. F., & Hayhoe, M. (2001) In what ways do eye movements contribute to everyday activities? Vision Research, 41 (25-26), 3559–3565.
Leider, F., & Griffiths, T. L. (2017). Strategy selection as rational metareasoning. Psychological Review, 124 (6), 762–794.
Malcolm, G. M., & Henderson, J. M. (2010). Combining top-down processes to guide eye movements during real-world scene search. Journal of Vision, 10 (2): 4, 1–11, https://doi.org/10.1167/10.2.4. [PubMed] [Article]
Michal, A. L., & Franconeri, S. L. (2017). Visual routines are associated with specific graph interpretations. Cognitive Research: Principles and Implications, 2, 20, 1–10.
Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24, 167–202.
Moher, J., & Song, J.-H. (2014). Perceptual decision processes flexibly adapt to avoid change-of-mind motor costs. Journal of Vision, 14 (8): 1, 1–13, https://doi.org/10.1167/14.8.1. [PubMed] [Article]
Monsell, S. (2003). Task switching. Trends in Cognitive Science, 7 (3), 134–140.
Najemnik, J., & Geisler, W. S. (2005). Optimal eye movement strategies in visual search. Nature, 434 (7031), 387–391.
Navalpakkam, V., Koch, C., Rangel, A., & Perona, P. (2010). Optimal reward harvesting in complex perceptual environments. Proceedings of the National Academy of Sciences, 107 (11), 5232–5237.
Neider, M. B., & Zelinsky, G. J. (2006). Scene context guides eye movements during visual search. Vision Research, 46 (5): 614–621.
Ottes, F. P., Van Gisbergen, J. A., & Eggermont, J. J. (1985). Latency dependence of colour-based target vs. nontarget discrimination by the saccadic system. Vision Research, 25 (6), 849–862.
Renninger, L. W., Verghese, P., & Coughlan, J. (2007). Where to look next? Eye movements reduce local uncertainty. Journal of Vision, 7 (3): 6, 1–17, https://doi.org/10.1167/7.3.6. [PubMed] [Article]
Ross, N. M., & Kowler, E. (2013). Eye movements while viewing narrated, captioned, and silent videos. Journal of Vision, 13 (4): 1, 1–1, https://doi.org/10.1167/13.4.1. [PubMed] [Article]
Sailer, U., Flanagan, J. R., & Johansson, R. S. (2005). Eye-hand coordination during learning of a novel visuomotor task. Journal of Neuroscience, 25 (39), 8833–8842.
Schall, J. D. (2004). On the role of frontal eye field in guiding attention and saccades. Vision Research 44 (12), 1453–1467.
Schnitzer, B. S., & Kowler, E. (2006). Eye movements during multiple readings of the same text. Vision Research, 46 (10), 1161–1132.
Semizer, Y., & Michel, M. M. (2017). Intrinsic position uncertainty impairs overt search performance. Journal of Vision, 17 (9): 13, 1–17, https://doi.org/10.1167/17.9.13. [PubMed] [Article]
Shah, P., & Freedman, E. G. (2009). Bar and line graph comprehension: An interaction of top-down and bottom-up processes. Topics in Cognitive Science, 1–19.
Shenhav, A., Botvinick, M. M., & Cohen, J. D. (2013) The expected value of control: An integrative theory of anterior cingulate cortex function. Neuron, 79, 217–240.
Shenhav, A., Musslick, S., Lieder, F., Kool, W., Griffiths, T. L., Cohen, J. D., & Botvinick, M. M. (2017). Toward a rational and mechanistic account of mental effort. Annual Review of Neuroscience, 40, 99–124.
Trommershäuser, J., Maloney, L. T., & Landy, M. S. (2008). Decision making, movement planning and statistical decision theory. Trends in Cognitive Science, 12 (8), 291–297.
Torralba, A., Oliva, A., Castelhano, M. S., & Henderson, J. M. (2006). Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological Review, 113 (4), 766–786.
Voss, J. L., Bridge, D. J., Cohen, N. J., & Walker, J. A. (2017). A closer look at the hippocampus and memory. Trends in Cognitive Science, 21 (8), 577–588.
Wang, J. X., Cohen, N. J., & Voss, J. L. (2015). Covert rapid action-memory simulation (CRAMS): A hypothesis of hippocampal-prefrontal interactions for adaptive behavior. Neurobiology of Learning and Memory, 117, 22–33.
Wang, H. C., & Pomplun, M. (2012). The attraction of visual attention to texts in real-world scenes. Journal of Vision, 12 (6): 26, 1–17, https://doi.org/10.1167/12.6.26. [PubMed] [Article]
Wolfe, J. M. (2013). When is it time to move to the next raspberry bush? Foraging rules in human visual search. Journal of Vision, 13(3), 10, 1–17, https://doi.org/10.1167/13.3.10. [PubMed] [Article]
Wolpert, D. M., & Landy, M. S. (2012). Motor control is decision making. Current Opinions in Neurobiology, 22 (6), 996–1003.
Wu, C.-C., Kwon, O. S., & Kowler, E. (2010). Fitts's Law and speed/accuracy trade-offs during sequences of saccades: Implications for strategies of saccadic planning. Vision Research, 50 (21), 2142–2157.
Wu, C.-C., & Kowler, E. (2013). Timing of saccadic eye movements during visual search for multiple targets. Journal of Vision, 13 (11): 11, 1–21, https://doi.org/10.1167/13.11.11. [PubMed] [Article]
Wu, C.-C., Wick, F. A., & Pomplun, M. (2014). Guidance of visual attention by semantic information in real-world scenes. Frontiers of Psychology, 5, 54.
Zhao, M., Gersch, T. M., Schnitzer, B. S., Dosher, B. A., & Kowler, E. (2012). Eye movements and attention: the role of pre-saccadic shifts of attention in perception, memory and the control of saccades. Vision Research, 74, 40–60.
Figure 1
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by attribute and values of the attributes are not in conflict. Text is redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Figure 1
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by attribute and values of the attributes are not in conflict. Text is redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Figure 2
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by product and values of the attributes are in conflict for the two products. Text is not redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Figure 2
 
Example of the stimulus display in the Simultaneous condition in which the bars are grouped by product and values of the attributes are in conflict for the two products. Text is not redundant with the graph. The font size of the text was increased for the purposes of this illustration.
Figure 3
 
Perceptual motor conditions. Left: Simultaneous: Graph and text appear side by side. Center: Button Press: Graph and text are centered, with the sequential appearance of each triggered via a button press. Right: Eye-Contingent: Graph and text appear side by side, with the sequential appearance of each triggered via a saccade into the corresponding area of the screen.
Figure 3
 
Perceptual motor conditions. Left: Simultaneous: Graph and text appear side by side. Center: Button Press: Graph and text are centered, with the sequential appearance of each triggered via a button press. Right: Eye-Contingent: Graph and text appear side by side, with the sequential appearance of each triggered via a saccade into the corresponding area of the screen.
Figure 4
 
Location of the boundaries of the seven AOI's in the display.
Figure 4
 
Location of the boundaries of the seven AOI's in the display.
Figure 5
 
Left: Mean number of visits/trial to graph and text (±SE). Center: Mean duration (in seconds) of visits to graph and text (±SE). Right: Mean visits/second (±SE). Data are shown for the three perceptual motor conditions and the two instruction types. Means are based on means of subject means, 11 subjects for each instruction. See Supplementary Tables S1 and S2 for individual subject data, and Supplementary Figure S2 for distributions of the number of visits/trial.
Figure 5
 
Left: Mean number of visits/trial to graph and text (±SE). Center: Mean duration (in seconds) of visits to graph and text (±SE). Right: Mean visits/second (±SE). Data are shown for the three perceptual motor conditions and the two instruction types. Means are based on means of subject means, 11 subjects for each instruction. See Supplementary Tables S1 and S2 for individual subject data, and Supplementary Figure S2 for distributions of the number of visits/trial.
Figure 6
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 1 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds to the duration of the visit (accumulation of fixations) to the graph or text. Arrows on the Simultaneous timelines show examples of trials with brief glances to graph or text at the beginning of the trial before a longer visit to the other region (see Analysis section). Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Figure 6
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 1 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds to the duration of the visit (accumulation of fixations) to the graph or text. Arrows on the Simultaneous timelines show examples of trials with brief glances to graph or text at the beginning of the trial before a longer visit to the other region (see Analysis section). Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Figure 7
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 2 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds the duration of the visit (accumulation of fixations) to the graph or text. Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Figure 7
 
Timelines depicting time spent viewing the graph (green), text (blue), or other areas (red) as the trial progressed for Instruction 2 for the Simultaneous (left), Button Press (center), and Eye-Contingent (right) conditions. Trials from the 11 subjects are ordered from shortest to longest. The length of each timeline bar corresponds the duration of the visit (accumulation of fixations) to the graph or text. Blank regions of the graph indicate that the visit prior to the blank contained a period of tracker lock lost greater than 2 s.
Figure 8
 
Histograms showing the durations of the first three visits for all trials in which the graph was visited first. The sequence was graph (top), text (middle), and graph again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
Figure 8
 
Histograms showing the durations of the first three visits for all trials in which the graph was visited first. The sequence was graph (top), text (middle), and graph again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
Figure 9
 
Histograms showing the durations of the first three visits for all trials in which the text was visited first. The sequence was text (top), graph (middle) and text again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
Figure 9
 
Histograms showing the durations of the first three visits for all trials in which the text was visited first. The sequence was text (top), graph (middle) and text again (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Mean duration, SD, and number of visits are shown on each graph.
Figure 10
 
Mean time (± SE) spent visiting new AOI's of the graph or new lines of text (blue) and re-examining previously visited AOI's or lines of text (red) for the first three visits and over the whole trial (All) for visits to the graph (top) and text (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Numbers above each bar represent the number of subjects that contributed to the means. A subject was not included if they did not have any second or third visits to graph or text, respectively, in that condition.
Figure 10
 
Mean time (± SE) spent visiting new AOI's of the graph or new lines of text (blue) and re-examining previously visited AOI's or lines of text (red) for the first three visits and over the whole trial (All) for visits to the graph (top) and text (bottom). Results are shown for the Simultaneous, Button Press, and Eye-Contingent conditions. Numbers above each bar represent the number of subjects that contributed to the means. A subject was not included if they did not have any second or third visits to graph or text, respectively, in that condition.
Figure 11
 
Matrices depicting the frequency of transitions between different areas of interest (AOI) within the graph or the text for all trials for each of the three perceptual motor conditions. The rows indicate the AOI of the previous visit and the columns indicate the AOI of the current visit. The color of each cell depicts the number of transitions from the AOI of the previous visit to the current visit. L = legend; YL = Y-axis label of left pair of bars; YR = Y-axis label of right pair of bars; BL = left pair of bars; BR = right pair of bars; X = X-axis label; T = title.
Figure 11
 
Matrices depicting the frequency of transitions between different areas of interest (AOI) within the graph or the text for all trials for each of the three perceptual motor conditions. The rows indicate the AOI of the previous visit and the columns indicate the AOI of the current visit. The color of each cell depicts the number of transitions from the AOI of the previous visit to the current visit. L = legend; YL = Y-axis label of left pair of bars; YR = Y-axis label of right pair of bars; BL = left pair of bars; BR = right pair of bars; X = X-axis label; T = title.
Table 1
 
Means of subject means of proportion of trials in which the text was visited first (for visits >1 s).
Table 1
 
Means of subject means of proportion of trials in which the text was visited first (for visits >1 s).
Supplement 1
×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×