Oculomotor dominance in multitasking: Mechanisms of conflict resolution in cross-modal action

Abstract

In daily life, eye movement control usually occurs in the context of concurrent action demands in other effector domains. However, little research has focused on understanding how such cross-modal action demands are coordinated, especially when conflicting information needs to be processed conjunctly in different action modalities. In two experiments, we address this issue by studying vocal responses in the context of spatially conflicting eye movements (Experiment 1) and in the context of spatially conflicting manual actions (Experiment 2, under controlled eye fixation conditions). Crucially, a comparison across experiments allows us to assess resource scheduling priorities among the three effector systems by comparing the same (vocal) response demands in the context of eye movements in contrast to manual responses. The results indicate that in situations involving response conflict, eye movements are prioritized over concurrent action demands in another effector system. This oculomotor dominance effect corroborates previous observations in the context of multiple action demands without spatial response conflict. Furthermore, and in line with recent theoretical accounts of parallel multiple action control, resource scheduling patterns appear to be flexibly adjustable based on the temporal proximity of the two actions that need to be performed.

Introduction

Visual orienting is typically characterized by regular switches between rapid movements of the eyes (saccades) and phases of relative rest (fixations; Findlay & Gilchrist, 2003; Rayner, 2009). In the past decades, the underlying oculomotor control processes have been thoroughly studied on the level of both their neural underpinnings and their cognitive foundations (e.g., Findlay & Walker, 1999; Hallett, 1978; Liversedge, Gilchrist, & Everling, 2011). However, most of these previous research efforts have focused on the control of eye movements in isolation, e.g., in reading (Kliegl, Nuthmann, & Engbert, 2006; Rayner, 1998) or in attention and perception processes (Kowler, 2011; Schütz, Braun, & Gegenfurtner, 2011). This is the case even though oculomotor control in daily life is clearly embedded into a vast array of simultaneous action demands in other motor domains, such as manual or vocal actions, e.g., during typewriting or reading aloud. In the present study, we address the issue of how oculomotor control interacts with concurrent motor control demands in other effector domains (i.e., cross-modal action, Huestegge & Hazeltine, 2011). 

Traditionally, processes of multiple simultaneous action demands are studied within the field of dual tasking, where, in a typical experiment, two speeded tasks need to be performed at the same time. Interestingly, many dual-task studies employ experimental paradigms in which responses in each task are executed within the same action modality (typically manual responses; see, e.g., Pashler, 1994). Considerably fewer studies involve cross-modal response demands, e.g., concurrent manual and vocal action (Hazeltine, Ruthruff, & Remington, 2006), and only very little attention has been paid to the study of other response modalities (especially eye movements). This lack of empirical evidence may have occurred for several reasons. First, the specific response modalities in terms of effector systems in dual tasks have often been regarded as irrelevant for the central cognitive operations of interest, due to their peripheral nature (e.g., Meyer & Kieras, 1997). Second, eye movements in psychological sciences have often been studied more as an indicator of visual attention (i.e., as a precondition for perceptual processes) than as an action modality. This particularly holds for studies that explicitly focused on the coordination of eye and hand movements in the context of reaching and grasping (e.g., Issen & Knill, 2012), where eye movements are supposed to provide visual feedback for optimized manual movement control (for a review, see Huestegge, 2011). Third, for a long time it has been assumed that saccades are special in that they are able to bypass any central response processing operations and thus are not subject to cross-response interference (Bekkering, Adam, Kingma, Huson, & Whiting, 1994; Pashler, Carrier, & Hoffman, 1993). However, more recent studies suggest that substantial interference can indeed occur when saccades need to be coordinated with concurrent response demands in other effector domains (e.g., Huestegge & Koch, 2009). Nevertheless, the underlying processing mechanisms of cross-modal response coordination involving oculomotor control have mainly remained unclear. 

Recently, a study (Huestegge & Koch, 2013) explicitly addressed cognitive processes underlying multiple action control across three response modalities (oculomotor, manual, vocal) and reported the first evidence for an oculomotor dominance effect. Specifically, participants responded to unilaterally presented auditory stimuli with a single response, or with two concurrent responses (in dual-response conditions), that were always spatially compatible to the stimuli. Across three experiments, three different response modalities were studied pairwise, i.e., saccades and manual responses, manual and vocal responses, and saccades and vocal responses. In each experiment, asymmetric dual-response costs were observed, insofar as response time differences between single- and dual-response conditions varied between response modalities. Crucially, this dual-response cost asymmetry was interpreted as an empirical marker for prioritization of response processing, in that the modality with smaller dual-response costs received more processing priority. Together, the pattern of cost asymmetries across experiments suggested an ordinal structure of priorities across response modalities: Saccades were prioritized over vocal and manual responses (oculomotor dominance effect), whereas vocal responses were prioritized over manual responses. Importantly, this dominance pattern could not be explained in terms of differences in single-response speed, e.g., in accordance with a first-come-first-served principle of priority scheduling as in response selection bottleneck theory (Pashler, 1994). The results rather appeared to represent modality-specific patterns of resource scheduling. It is notable that the observed oculomotor dominance effect on the output side of processing resembles similar visual dominance effects on the input side of processing, where visual stimuli are usually processed with greater priority than auditory stimuli when both are presented at the same time (Colavita, 1974; Posner, Nissen, & Klein, 1976; Spence, 2009). Thus, the occurrence of oculomotor dominance complements the interpretation of the well-established visual dominance effect by suggesting that the visual system in principle (including both input and output processing) dominates other concurrent processing demands. 

However, some aspects of the study by Huestegge and Koch (2013) limit the generalizability of the postulated dominance scheme. On a general level, it is unclear to what extent the results pattern may depend on the specific paradigm and conditions used. More specifically, responses in dual-response conditions were always spatially compatible, suggesting that there were no reasonable conflict resolution demands involved. However, any prioritization mechanisms regarding resource scheduling among effector systems may be considered especially important in situations that involve spatial conflict between the required action demands, e.g., typing in numbers on the number pad located at the right side of a keyboard while at the same time looking at the resulting changes on the left part of the screen. Furthermore, it is well known from dual-task studies utilizing the psychological refractory period (PRP) paradigm that the temporal distance by which two responses are separated in an experimental trial determines the amount of between-task interference and therefore the observed dual-response costs (Pashler, 1984; Pashler & Johnston, 1989). Typically, the reaction time of the later response is slower when both responses are executed at smaller temporal distances, while the same reaction is much faster when the executions of the two responses are further apart (known as the PRP effect; Pashler, 1994). Therefore, it should be particularly interesting to study modality dominance effects in a paradigm that (a) involves conflict, e.g., in terms of spatially incompatible response demands, and (b) also allows for temporal response distance manipulations. 

In another previous study, Huestegge and Koch (2010) introduced a paradigm that appears ideally suited to address the particular issue of response distance manipulation, namely the crossed-incompatibility paradigm (CIP). This paradigm is comparable to the one employed by Huestegge and Koch (2013, see previously), but involves spatially incompatible instead of compatible responses. For example, an auditory stimulus in the left ear requires two simultaneous responses, a left (i.e., spatially compatible) saccade and a right (i.e., spatially incompatible) manual response. In another group of participants, the reversed assignment is implemented (i.e., an incompatible saccade and a compatible manual response). Crucially, these reversed stimulus-response (S-R) assignments cause responses to be temporally close together in one group and more distant from each other in the other group, while the fact that both responses are spatially incompatible with each other (i.e., the spatial response-response conflict) remains the same in both temporal distance groups. However, this study was restricted to the combination of saccades and manual responses only, and was not explicitly designed to address the issue of response modality dominance. While the observed asymmetry of dual-response costs (i.e., smaller dual-response costs for saccades than for manual responses) could be interpreted in terms of an oculomotor dominance effect, the two other combinations of response modalities (saccades and vocal responses, manual and vocal responses) were lacking. Consequently, it was not possible to come up with conclusive results regarding response modality dominance patterns in situations involving response conflict. 

In the present study, we intended to make the following novel contributions to the issue of eye movements and response modality dominance: First, we aimed at broadening the implications of the work of Huestegge and Koch (2013) by introducing cross-response conflict while still utilizing dual-response cost asymmetries to derive priority assignments. Specifically, in Experiment 1 we employed the CIP to study the simultaneous execution of spatially incompatible oculomotor responses and vocal responses. Note that the combination of oculomotor responses and manual responses within the CIP was already studied previously (Huestegge & Koch, 2010). Second, we aimed at comparing these results from Experiment 1 with those of Experiment 2, where we combined vocal and manual responses under controlled fixation conditions. Experiment 2 is necessarily important in order to obtain a comprehensive view of prioritization patterns among all three effector systems (oculomotor, vocal, and manual), because only a comparison across experiments allows us to assess resource scheduling priorities among the three effector systems. This can be achieved by comparing the same (here: vocal) response demands in the context of eye movements versus manual responses; a complete prioritization pattern among modalities cannot be concluded from one single experiment alone. Note that prior studies on conflict resolution in dual-response control, which typically have combined vocal and manual responses as in the present Experiment 2, have never controlled for the occurrence of eye movements (see also Huestegge & Hazeltine, 2011). This has probably led to an additional source of response interference based on saccade execution, which naturally occurs whenever visuospatial stimulation is utilized. Finally, the opportunity to manipulate temporal response distance within the CIP allows us to additionally vary the potential for response conflict, since a temporally close execution of incompatible responses should be associated with greater conflict potential compared to responses that are executed with a substantial temporal delay. Consequently, this manipulation might also imply shifts of priority patterns. 

More specifically, three potential theoretical scenarios are in play: (a) Rigid modality-independent resource scheduling. If dual-response cost asymmetries were based on an a-modal, inflexible first-come-first-served mechanism (e.g., similar to that in response selection bottleneck theory, see Pashler, 1994), the slower response in each dual-response combination should always exhibit larger dual-response costs. (b) Strict modality-based prioritization. If, regardless of the specific combination of response modalities and regardless of the amount of conflict present, specific effector modalities were consistently preferred over others, we would expect a dual-response cost priority pattern similar to that seen by Huestegge and Koch (2013), but without any modulation as a function of the potential for conflict. (c) Flexible modality-based prioritization. If resource scheduling were affected by the potential for conflict (i.e., by temporal response distance), this should become evident in additional significant shifts of the dual-response cost pattern as a function of response distance. Furthermore, the manipulation of response distance in both experiments can also be informative regarding the issue of serial versus parallel response processing in the context of cross-modal dual-response demands (see Huestegge & Koch, 2010, and the final part of the General discussion). 

Experiment 1

Experiment 1 was designed to test the prioritization hypotheses by comparing dual-response costs of simultaneously executed saccades and vocal responses within the CIP. Thus, the overall method (i.e., the CIP rationale) is largely comparable to the procedure described by Huestegge and Koch (2010). For example, a left auditory stimulus in dual-response conditions required two simultaneous responses, namely a left (spatially compatible) saccade and a “right” (spatially incompatible) vocal response (i.e., uttering the word “right”). This particular S-R assignment causes temporally distant responses, since the (already generally slower) vocal response is further delayed through the spatially incompatible S-R rule. In another group of participants, we implemented the reverse S-R assignment, i.e., an incompatible saccade and a compatible vocal response. This assignment caused both responses to be executed temporally closer to each other, while the fact that both responses were spatially incompatible with each other (i.e., the spatial response-response conflict) remained the same in both temporal distance groups. Additionally, both groups were tested in single-response blocks to calculate dual-response costs, defined as the difference between single-response and dual-response performance. 

Method

Participants

Twenty-four participants were randomly assigned to two groups (equivalent to the two response distance conditions). The mean age amounted to 22.5 years (SD = 3.0) in the distant-responses group (11 female and one male) and 22.4 years (SD = 3.2) in the close-responses group (nine female and three male). All participants were naïve regarding the purpose of the experiment and received either course credits or reimbursement for participation. The research protocol of the experiment adhered to the tenets of the Declaration of Helsinki. 

Stimuli and apparatus

Participants were seated in front of a 20-in. cathode-ray tube monitor (temporal resolution: 100 Hz; spatial resolution: 1024 × 768 pixels) with a keyboard in front of them. A chin rest was installed at a distance of 67 cm from the monitor to control for major head movements. A microphone to record the participants' vocal responses was placed at a distance of approximately 10 cm from the chin rest. Vocal latencies were defined as the time from stimulus onset until the sound pressure of an utterance exceeded a predefined threshold (voice key procedure implemented in the experiment presentation software ExperimentBuilder) that was determined in a pilot experiment.¹ 

Eye movements were registered using an EyeLink II system (SR Research, Ottawa, Canada) by measuring the position of the right pupil with a temporal resolution of 500 Hz and a spatial resolution of less than 0.01° (EyeLink II pupil only mode). Saccade latencies were defined as the interval between stimulus onset and the initiation of the saccade using the built-in saccade parser (velocity threshold = 30 °/s; acceleration threshold = 8000 °/s²). The space bar of the keyboard was used during calibration routines (horizontal three-point calibration). 

On the screen, a green central fixation cross (on a black background) was flanked by two green rectangles that served as saccade targets at an 8° visual angle to the left and right and were permanently present during an experimental block. The size of both the fixation cross and the rectangular saccade targets amounted to 0.33° each. Auditory stimuli consisted of unilateral 1000-Hz pure tones (easily audible, duration: 50 ms) and were presented via supra-aural headphones. 

Procedure

In each trial, the imperative auditory stimulus was presented either to the left or right ear in random order. In the distant-responses group, participants were instructed to move their gaze to the spatially compatible rectangle (saccade single-response blocks), to utter the spatially incompatible word “links” (“left”) or “rechts” (“right”; vocal single-response blocks), or to perform both responses simultaneously (dual-response blocks). Participants in the close-responses group were instructed with inverted S-R mappings (i.e., S-R incompatible gaze shifts and S-R compatible vocal responses, see Figure 1). Participants were asked to execute responses as fast and accurately as possible without any instructions about the response order. 

Figure 1

 

Crossed-incompatibility paradigm (CIP): Trial structure (left) and corresponding manipulation of temporal response distance (right) in the two response distance groups in Experiment 1.

In trials requiring a saccade reaction, participants were told to return to the central fixation cross afterwards, while in vocal single-response trials the gaze was to remain on the central fixation cross. Each participant completed nine blocks consisting of 30 trials each (three blocks for each of the three block types). At the beginning of each block, the eye-tracking system was calibrated. 

Design

The design consisted of three independent variables, namely response modality (saccade and vocal response), response condition (single and dual), and response distance (close and distant). While the former two variables were manipulated within subjects, response distance was a group factor. The order of the three block types (two single-response blocks and one dual-response block) was counterbalanced across participants. Response times (RTs) and errors for saccades and vocal responses were measured as dependent variables. 

Results and discussion

Correct saccades required a minimal amplitude of 4° (half of the visual angle of the lateral saccade targets) in the instructed direction, as indicated by a landing position on the left or right of the central fixation cross. Saccades in vocal single-response trials with an amplitude larger than 2° were defined as erroneously executed. We excluded trials with anticipatory responses (saccade RTs < 70 ms and vocal RTs < 150 ms), trials involving technical malfunctions, and trials in vocal single-response conditions with erroneously executed eye movements. Taken together, this procedure resulted in 95.9% valid data. Furthermore, we excluded trials in which the vocal response was executed prior to the saccade (0.2% of the valid data). 

For the RT analysis, only data from correct trials were submitted to a mixed three-way ANOVA. Figure 2 shows mean RTs of saccades and vocal responses as a function of response condition and response distance. Statistical analyses revealed a significant main effect of modality, F(1, 22) = 639.9, p < 0.001, η²_p = .97, indicating the typical observation that vocal RTs (753 ms) are longer than saccade RTs (287 ms). The main effect of response condition was significant, too, F(1,22) = 61, p < .001, η²_p = 0.74, indicating overall dual-response costs of 100 ms. We did not observe a significant main effect of response distance, F(1, 22) = 2.8, p > 0.10, suggesting that overall mean RTs did not differ substantially between groups. This observation is nicely in line with the fact that the two responses were spatially incompatible with each other (and thus producing a similar amount of crosstalk) in both temporal distance conditions. 

Figure 2

 

(A) Mean response times (RTs) for vocal responses and saccades as a function of response condition (single and dual) and response distance (distant and close) in Experiment 1. (B) Dual-response costs (in milliseconds) for vocal responses and saccades as a function of S-R compatibility in both response distance groups. (C) Proportional dual-response costs (in percentages) for vocal responses and saccades as a function of S-R compatibility in both response distance groups. Error bars represent standard errors.

The interaction of response modality and response condition, F(1, 22) = 31.7, p < 0.001, η²_p = 0.59, indicates significantly larger dual-response costs for vocal responses (167 ms) than for saccades (32 ms), which is in line with prior observations of asymmetrical costs across modalities (Huestegge & Koch, 2010; 2013). Post hoc comparisons revealed that dual-response costs for vocal responses were larger than for saccades in both response distance conditions (127 ms vs. 45 ms in the distant-responses group, t = 2.9, p = 0.015; and 207 vs. 18 ms in the close-responses group, t = 4.9, p < 0.001). Overall, this pattern is in line with the assumption of an oculomotor dominance effect in that performance costs for saccades were always smaller than for vocal responses. 

The interaction of response modality and response distance, F(1, 22) = 16.6, p = 0.001, η²_p = 0.43, shows that responses in the close-responses group were indeed executed significantly closer to each other (with a difference of 390 ms) than in the distant-responses group (where responses were executed with a mean distance of 540 ms). This demonstrates the effectiveness of our response distance manipulation. 

Interestingly, the interaction of response condition and response distance was far from being statistically significant, F(1, 22) = 1.1, p > 0.30, suggesting that dual-response costs were not significantly affected by response distance. While this result resembles those of Huestegge and Koch (2010), where dual-response costs were equal for each modality in both response conditions, it may appear surprising, since previous research on multiple action control has usually reported greater dual-response costs when two responses need to be executed at close temporal proximity (e.g., Pashler, 1994). 

Importantly, in addition we found a significant three-way interaction, F(1, 22) = 4.8, p = 0.039, η²_p = 0.18, suggesting that dual-response costs for saccades decreased from the distant to close response conditions, while dual-response costs for vocal responses increased (see Figure 2B). This three-way interaction suggests that processing priorities among the two response modalities are shifted from one modality to the other across the two response distance conditions, indicating flexible resource scheduling. Specifically, it appears as if costs in one response modality have been compensated for by a relative benefit in the other modality, indicating a trade-off between common limited resources across response modalities. Probably, when saccades need to be performed in an S-R incompatible and thus more difficult way (similar to an antisaccade task; see Hallett, 1978) in the close-responses conditions, they are prioritized higher than usual (i.e., compared to being S-R compatible), so that the remaining (vocal) response modality shows relatively larger dual-response costs. Taken all together, this observation is clearly in line with the assumption of an adjustable modality-dependent processing prioritization. 

Because of the large single-response RT difference between response distance groups, and based on the fact that modalities differ in their absolute RT level, we additionally analyzed proportional (instead of absolute) dual-response costs (in percentages) in order to control for any effects that might be caused simply by baseline (i.e., single RT level) differences. Proportional costs were computed for each individual participant ((dual RT − single RT)/single RT) and then submitted to a 2×2 ANOVA with factors of response modality and response distance. The analysis revealed the same statistical pattern as the analysis of absolute RTs: Proportional dual-response costs were significantly greater for vocal responses (25%) than for saccades (14%), F(1, 22) = 5.8, p = 0.025, η²_p = 0.21, while there was no significant difference between the response distance conditions, F < 1. Importantly, the interaction of modality and response distance was again significant, F(1, 22) = 8.3, p = 0.009, η²_p = 0.27, confirming the cost trade-off pattern across modalities as a function of response distance, equivalent to that observed in absolute dual-response costs (see the three-way interaction for absolute RTs). 

To check for response accuracy, we also analyzed errors, although the overall error rate was very low and amounted only to 3.8%. A mixed three-way ANOVA on the error data revealed a significant main effect of response modality, F(1, 22) = 14.6, p = 0.001, η²_p = 0.40, demonstrating the usual finding that saccades exhibit more errors (5.8%) than vocal responses (1.8%). Interestingly, the main effect of response distance was significant too, F(1, 22) = 8.9, p = 0.007, η²_p = 0.29, indicating that more errors were committed in conditions with close responses (6.2%) than in conditions with distant responses (2.4%). Thus, executing two responses at a close temporal distance appeared to be more difficult than executing distant responses. This observation might be due to the fact that the execution of incompatible saccades (which is only necessary in close-responses conditions) is particularly difficult. In line with this interpretation, we found a significant interaction of response modality and response distance, F(1, 22) = 6.4, p = 0.019, η²_p = 0.27, indicating that the difference in saccade errors between the distant and close response conditions (2.1% vs. 9.5%) was much larger than that for vocal responses (0.7% vs. 2.9%). We did not find a significant main effect of response condition, F(1, 22) = 2.9, p > 0.10. Finally, there was no significant interaction of response distance and response condition, F(1, 22) = 2.1, p > 0.15, no interaction of response modality and response condition, F < 1, and no three-way interaction, F < 1. 

Taken together, the results from this experiment strengthen the assumption of an oculomotor dominance effect (Huestegge & Koch, 2013), which appears to be also present in situations involving response conflict. However, the present results also offer novel insight in that modality-specific resource allocation across effector systems can be flexibly adjusted based on the temporal distance (and thus, the potential for conflict) between the two responses. 

Based on the data from the (slower) vocal responses alone, one might at first conclude that the increase of vocal dual-response costs in the close-responses conditions appears to be quite in line with a first-come-first-served principle according to a serial processing account, which explicitly predicts that under close response processing conditions, the processing of the second response is delayed until response selection for the first response is finished. However, the corresponding reversed pattern of dual-response costs for saccades is clearly at odds with a serial processing account, since dual-response costs for saccades even increase with a larger response distance. Thus, this result pattern rather indicates a trade-off of response assignments across response modalities. 

Unfortunately, the data from this experiment alone do not allow us to derive a conclusive assessment of prioritization patterns among effector modalities, since we cannot compare the dual-response costs in the present experiment with similar conditions involving other combinations of response modalities. To address this issue, we conducted Experiment 2, which involves the simultaneous execution of manual and vocal responses, but under controlled conditions regarding the occurrence of eye movements. Even though Experiment 2 does not explicitly involve the execution of eye movements, a comparison of data patterns across experiments will enable us to indirectly infer the impact of contextual responses on the prioritization pattern observed in Experiment 1, and thus it can be informative regarding prioritization patterns with respect to eye movements. 

Experiment 2

Experiment 2 addresses the issue of multiple response control in combined vocal and manual responses within the CIP under controlled eye-movement demands (i.e., instructions to remain fixated). The main goal is to be able to interpret the prioritization pattern of Experiment 1 more conclusively by studying one of the two response modalities from that experiment (i.e., vocal response) again, but in the context of another (here: manual) response modality. Apart from the replacement of saccades with manual key-press responses, all other methodological aspects are the same as in Experiment 1. 

Method

Participants

A new sample of 24 participants (21 female, three male) was randomly assigned to the two response distance groups. The mean age amounted to 21.4 years (SD = 2.7) in the distant-responses group (11 female, one male) and 22.6 years (SD = 3.2) in the close-responses group (10 female, two male). They were naïve regarding the purpose of the experiment and received either course credits or reimbursement for participation. The research protocol of the experiment adhered to the tenets of the Declaration of Helsinki. 

Stimuli and apparatus

The hardware setup was identical to that in Experiment 1. Additionally, the keyboard was used to record manual key-press responses. Two keys (left Ctrl and right arrow) served as response keys and were operated by participants' left and right index fingers. The eye-tracking system was utilized to control for the occurrence of eye movements. 

Procedure and design

The overall procedure and design were the same as in Experiment 1 except for the fact that manual responses were required instead of saccade responses. Instructions for the vocal task were the same as in Experiment 1. In conditions involving a manual response, participants in the distant-responses group were instructed to press the spatially compatible key. Conversely, participants in the close-responses group were asked to press the spatially incompatible key. Throughout the experiment, participants were instructed to keep their gaze on the central fixation cross. 

Results and discussion

We excluded trials with anticipatory responses (RTs < 150 ms in both effector modalities) and trials with erroneously executed eye movements (saccade error defined as in Experiment 1), which resulted in 95.5% valid data. Furthermore, we excluded trials in which the vocal response was executed prior to the manual response (1.4% of the valid data). 

For the RT analysis, only data from correct trials were submitted to a mixed three-way ANOVA. Figure 3 shows manual and vocal RTs as a function of response distance and response condition. The main effect of response modality was significant, F(1, 22) = 254.5, p < 0.001, η²_p = 0.92, showing that vocal responses (721 ms) were slower than manual responses (413 ms). The main effect of response condition was significant too, F(1, 22) = 105.1, p < 0.001, η²_p = 0.83, indicating longer RTs in dual-response conditions (618 ms) than in single-response conditions (515 ms). There was no significant main effect of response distance, F < 1. 

Figure 3

 

(A) Mean response times (RTs) for vocal and manual responses as a function of response condition (single and dual) and response distance (distant and close) in Experiment 2. (B) Dual-response costs (in milliseconds) for vocal and manual responses as a function of S-R compatibility in both response distance groups. (C) Proportional dual-response costs (in percentages) for vocal and manual responses as a function of S-R compatibility in both response distance groups. Error bars represent standard errors.

We observed a significant interaction of response modality and response distance, F(1, 22) = 8.1, p = 0.009, η²_p = 0.27, indicating that responses in the close-responses group were indeed executed significantly closer to each other (difference of 254 ms) than those in the distant-responses group (difference of 363 ms). Again, this indicates that our temporal response distance manipulation was successful. 

Importantly, there was no significant interaction of response condition and response distance, F < 1. Thus, similar to Experiment 1, there was no robust indication of a difference in overall dual-response costs between the two response distance conditions. The interaction of response condition and response modality was not significant either, suggesting that there was no reliable difference in dual-response costs between modalities, F < 1, and thus no clear prioritization of one modality over the other. This observation differs from the data by Huestegge and Koch (2013), where vocal responses exhibited significantly smaller dual-response costs than manual responses in an experimental paradigm without cross-response conflict. 

The three-way interaction was marginally significant, F(1, 22) = 3.4, p = 0.078, η²_p = 0.13, resembling the observed trend in Experiment 1 (see Figure 3B): Dual-response costs for vocal responses (i.e., the slower response) tended to be larger when responses were executed closer together compared to the distant-responses condition (129 ms vs. 79 ms), while manual dual-response costs show a reversed pattern (95 ms vs. 106 ms). Thus, similar to the corresponding results in Experiment 1, we observed a tendency towards a trade-off between modality-based resources as a function of response distance, indicating a certain degree of flexibility in resource scheduling. 

Given the substantial difference in overall RT levels across response modalities, we again computed proportional dual-response costs for each participant in both modalities and submitted them to an ANOVA with the independent variables of response modality and response distance. This analysis revealed no significant main effect of response distance, F < 1. However, unlike in the absolute RT data, here we found a significant main effect of response modality, F(1, 22) = 15.3, p < 0.001, η²_p = 0.41, indicating larger proportional dual-response costs for manual responses (29%) than for vocal responses (16%), which can be interpreted in terms of a prioritization of the vocal response (compare Huestegge & Koch, 2013). Importantly, this result is further qualified by a significant interaction of response modality and response distance, F(1, 22) = 6.3, p = 0.020, η²_p = 0.22, revealing that proportional dual-response costs of the (slower, in terms of single RT speed) vocal response were larger for close-responses condition (21%) than for distant-responses condition (11%), while dual-response costs for manual responses exhibited a reversed pattern (26% vs. 33%). This result further substantiates the claim that central resources were strategically allocated across response modalities, in that the conflict-affected S-R incompatible response tended to be (relatively) prioritized over the S-R compatible response. 

The mean rate of errors only amounted to 1.6%. A corresponding ANOVA revealed no statistically significant main effects or interactions. 

It should be noted that the “remain fixated” instructions in the present experiment might have slightly increased the overall cognitive burden, due to the corresponding inhibitory control demands. However, given that these demands persisted throughout all conditions of the present experiment, this may have increased the overall RT level but should not have compromised any of the critical patterns of results reported here. 

Although Experiment 1 might have suggested a serial processing account in which the faster response is processed with a higher priority than the slower one, the data of Experiment 2 clearly demonstrate the contrary. Here, the faster manual response exhibits at least the same amount of dual-response cost as the slower vocal response, thus suggesting instead a parallel processing mechanism (see also General discussion). Nevertheless, especially with respect to the analysis of proportional dual-response costs, the results of Experiment 2 are again in line with the assumption of flexible resource adjustment modulated by the potential for conflict, and indicate that the cognitive mechanisms proposed in Experiment 1 may not only hold for the special case of combined saccades and vocal responses but rather represent cross-modal action control mechanisms in general. 

Comparison across experiments

Experiment 1 provided strong evidence that oculomotor responses are prioritized over vocal responses, while overall analyses of Experiment 2 suggest that vocal responses show the tendency to be prioritized over manual responses. This ordinal prioritization pattern derived from situations involving response conflict nicely converges with previous observations in situations without such conflict (Huestegge & Koch, 2013). However, a comparison of data across experiments is necessary to check if saccades are also prioritized over manual responses, and to test the hypothesis (see Introduction) that resource allocation for a specific response modality depends on the contextual response demand, not only in terms of its proximity or S-R compatibility but also in terms of its effector modality. 

In order to compare the results of Experiments 1 and 2, we applied the same logic for the interpretation of dual-response cost asymmetries as before, insofar as the cost asymmetry served as a marker for prioritization, i.e., lower costs represent a higher priority. For determining a comprehensive priority pattern among modalities, we compared dual-response costs of the same modality (vocal responses) in two different contexts, here the two other modalities (saccade and manual response). The difference in dual-response costs for the vocal modality between the two context response conditions tells us which context modality is dominant in terms of the amount of dual-response cost in the vocal response. Directly comparing dual-response costs of the context responses is additionally applied to corroborate the priority pattern. 

To ensure that RT levels of the vocal response are comparable and did not differ between experiments due to interindividual differences, we calculated a cross-experiment ANOVA on single vocal response RTs as the dependent variable and experiment (1 vs. 2) and S-R compatibility² (compatible vs. incompatible) as independent variables. Crucially, the main effect of experimental session was not significant, F < 1, p = 0.98, providing no evidence for an overall difference in single vocal RTs (mean of 669 ms in both experiments). The main effect of S-R compatibility was significant, F(1, 44) = 7.16, p = 0.01, η²_p = 0.14, which shows that S-R incompatible responses (706 ms) took longer than compatible responses (632 ms; compatibility effect). The interaction was not significant, F < 1, p = 0.90, indicating that the compatibility effect did not differ between experiments. Taken together, absolute vocal RTs were almost identical across both experiments, providing a valid basis for further comparisons. 

In order to compare the amount of vocal dual-response costs between Experiments 1 and 2, we conducted a cross-experiment ANOVA for vocal dual-response costs. In addition, we computed a similar, separate analysis comparing dual-response costs of the context responses (i.e., saccades in Experiment 1 and manual responses in Experiment 2). Figure 4 shows the dual-response costs of vocal responses (black bars) in both experiments in form of absolute (A) and relative (B) RT differences. Note that in Figure 4 we averaged across the two response distance groups. 

Figure 4

 

Dual-response costs (in milliseconds) for vocal responses in the context of saccades (black bars) and manual responses (striped bars), and dual-response costs for saccades (Experiment 1) and manual responses (Experiment 2) in the context of the same (i.e., vocal) response demands. (A) Absolute dual-response costs. (B) Proportional dual-response costs. Error bars represent standard errors.

The 2×2 ANOVA for dual-response costs of vocal responses with the independent variables of context response modality (saccades in Experiment 1 and manual responses in Experiment 2) and response distance (close and distant) revealed a main effect of the context modality, F(1, 44) = 5.3, p = 0.026, η²_p = 0.11, indicating that dual-response costs for the same vocal responses were 62 ms larger when they were combined with saccades (167 ms) than with manual responses (105 ms). This finding shows that vocal costs are strongly determined by the identity of the context response, in that a vocal–saccade combination gives rise to greater vocal costs compared to a vocal–manual combination. This finding indicates an oculomotor dominance over manual responses. The main effect of response distance was significant too, F(1, 22) = 5.7, p = 0.021, η²_p = 0.12, showing that vocal dual-response costs were larger in the close-responses conditions (168 ms) than in the distant-responses conditions (104 ms). This result supports the observations from Experiments 1 and 2 that the relatively faster (S-R compatible) vocal response (compared to the S-R incompatible vocal response) suffers more dual-response interference. However, the two-way interaction was not significant, F < 1, indicating that the main effect of response distance was simply additive to the main effect of the context response modality. 

The ANOVA on dual-response costs of the context response modality with the independent variables of response modality and response distance also revealed a main effect of modality, F(1, 44) = 22.3, p < 0.001, η²_p = 0.34, indicating that saccades exhibited smaller costs (32 ms) than manual responses (101 ms) when executed simultaneously with vocal responses. This observation again provides evidence that saccades are prioritized over manual responses. There was no significant main effect of response distance, F(1, 44) = 1.6, p = 0.212, and no significant interaction, F < 1. 

Due to the large difference in absolute RT levels between saccades and manual responses, we computed an additional analysis of proportional dual-response costs across the two experiments. The ANOVA revealed a significant main effect of response modality, F(1, 44) = 11.3, p = 0.002, η²_p = 0.20, confirming that saccades exhibited smaller costs (14%) than manual responses (30%) when combined with identical responses in the vocal modality. In this analysis, the main effect of response distance was significant too, F(1, 44) = 5.1, p = 0.028, η²_p = 0.10, indicating larger proportional costs in the close-responses condition (27%) than in the distant-responses condition (16%). 

Taken together, the modulation of the amount of vocal dual-response costs as a function of the context response is direct evidence for the claim that resource scheduling for a specific response modality flexibly depends on the identity of the context response modality (Huestegge & Koch, 2013). When combined with manual responses instead of saccades, the same vocal response appears to suffer less, indicating that vocal responses receive relatively more processing resources. This is again in line with the assumption that vocal responses are prioritized over manual responses. Also, the finding that vocal responses exhibited much larger costs when combined with saccades than with manual responses reflects an ordinal prioritization structure in which saccades are strongly prioritized over vocal responses. The cross-experiment comparison of saccades and manual responses (in the context of comparable vocal contextual demands) again corroborates the assumption of oculomotor dominance. 

General discussion

The present study was designed to investigate the coordination of eye movements in the context of concurrent action demands in other effector domains, i.e., response modalities. Previous evidence for an oculomotor dominance effect—i.e., a prioritization of eye movement control in the context of other response demands—has been limited to situations without any potential for cognitive conflict (Huestegge & Koch, 2013). However, we reasoned that any prioritization mechanism regarding resource scheduling among effector systems should be particularly important in situations that involve strong conflict between action demands. Therefore, we studied modality dominance effects in a paradigm with spatially conflicting response demands by utilizing the crossed-incompatibility paradigm (CIP; Huestegge & Koch, 2010), which involves the execution of two spatially incompatible responses (in two different modalities) triggered by a unilateral auditory stimulus. While the results of Huestegge and Koch (2010), who combined saccades and manual responses, already indicated oculomotor prioritization in terms of smaller dual-response costs for saccades than for manual responses, their study was not specifically designed to investigate response modality dominance. With this particular research question in mind, the present study provides the “missing” response combinations (oculomotor and vocal responses, vocal and manual responses) needed to come up with conclusive results regarding effector system dominance during response conflict. The CIP also provides a manipulation of the potential for response conflict by varying temporal response proximity. In Experiment 1, we combined oculomotor and vocal responses within a group with temporally close responses and within a group with temporally distant responses. In Experiment 2, we employed the same paradigm but combined manual and vocal responses to be able to compare the pattern of dual-response costs across experiments. 

Flexible modality-based prioritization

Across both experimental groups, the results from Experiment 1 revealed clear evidence for an oculomotor dominance effect in terms of smaller dual-response costs for saccades than for vocal responses. This finding is in line with similar previous results within a paradigm involving spatially compatible responses only. Therefore, the present results suggest that resource scheduling mechanisms generalize to different experimental paradigms, i.e., they are also effective when cross-response conflict (Huestegge & Koch, 2009) is present. The observation of an oculomotor dominance effect in particular is strikingly similar to visual dominance effects reported in research on cross-modal attention (Colavita, 1974; Posner et al., 1976; Spence, 2009). From a more global, functional perspective, the prioritization of the visual system on both the input and output side of information processing may be considered helpful in detecting (or looking for) important (e.g., life-threatening) environmental changes, representing a prerequisite to any subsequent action involving other response modalities (e.g., calling for help). 

Another interesting issue is the comparison of the pattern of dual-response costs across experiments, because it allows us to assess resource scheduling priorities among the three effector systems. Two major conclusions can be drawn: First, the comparison of dual-response costs for saccades from Experiment 1 and for manual responses from Experiment 2 clearly confirmed smaller costs for saccades, again supporting the assumption of an oculomotor dominance effect. Second, we found a strong modulation of the amount of dual-response costs for the same vocal response demand across experiments. This highlights the flexible nature of resource scheduling with respect to action modalities, apparently depending on the specific requirements (here: effector modality) of the context response. The specific data pattern also suggests an oculomotor dominance in that vocal responses suffer more in the context of (prioritized) saccades compared to (less prioritized) manual responses. Additionally, vocal responses appear to dominate manual responses. Although the absolute RT data in Experiment 2 did not provide evidence for a vocal dominance over manual, three observations clearly support this assumption, namely the data from the proportional analyses in Experiment 2, the analyses from the comparison across experiments, and corresponding (both absolute and proportional) data from a previous study on cross-modal response control without response conflict (Huestegge & Koch, 2013). 

Furthermore, the results regarding the manipulation of temporal distance between the two simultaneous responses (i.e., the group comparison within each experiment) indicate that resource scheduling patterns are flexibly adjustable contingent upon the temporal proximity of the two actions that need to be performed. While the overall amount of dual-response costs was comparable across groups, closer responses (presumably associated with greater overall conflict between responses) led to a significant shift in the resource allocation pattern. Specifically, relatively more resources were allocated towards the response demand with the more difficult (i.e., incompatible) S-R assignment (compared to the S-R compatible response of the same modality). This assumption of processing dependency indicates that the two response-processing demands are not perfectly shielded against each other. It has been argued that the execution of one task can—under certain conditions—be shielded against distraction from other ongoing processing demands (e.g., Dreisbach & Wenke, 2011; Fischer, Gottschalk, & Dreisbach, 2013). Despite the lack of perfect shielding here, this theoretical framework generally fits into our interpretation of flexible resource scheduling. For example, it is possible that in conditions with temporally close responses, the S-R incompatible response is shielded in order to provide optimal performance, so that a relatively larger portion of dual-response costs is strategically transferred to the (easier) S-R compatible response without changing the overall effector-based response dominance pattern. Additionally, the observation that the costs of the dominant (oculomotor) response decrease from distant to close in Experiment 1 while the costs of the dominant vocal response increase in Experiment 2 implies that S-R compatibility is able to override the beneficial effects of response modality dominance. That S-R incompatible saccades dominate vocal responses even more than compatible ones in Experiment 1, together with the lack of a clear vocal dominance effect in absolute RTs in Experiment 2, may indicate that the vocal-over-manual dominance is less pronounced than the oculomotor-over-vocal dominance. Thus, although the general modality dominance pattern seems to be strong and hardly alterable, the actual strength of the modality prioritization appears to be variable and dependent on the specific response combination. 

The observation of strategic shifts of resource allocation corresponds with current theories of multiple action control that assume flexible resource scheduling among parallel response requirements (Lehle & Hübner, 2009; Meyer & Kieras, 1997; Navon & Miller, 2002; Tombu & Jolicoeur, 2003). For example, a specific computational model of resource scheduling in task-set control is ECTVA (Logan & Gordon, 2001), which involves the specification of sets of control parameters. While ECTVA has not explicitly specified response modality weighting mechanisms yet, this could probably be incorporated in the model (see Huestegge & Koch, 2013, for a more in-depth discussion). 

Parallel versus serial processing and oculomotor control

Note that the theoretical frameworks mentioned (e.g., parallel resource allocation, shielding) have in common that they presuppose the possibility of parallel selection and processing of responses, which stands in direct contrast to other frameworks that assume strictly serial response selection operations. Interestingly, the issue of parallel or serial processing of attentional processes has been intensively debated over the last 30 year in the vision literature, especially within the fields of visual search (e.g., Nakayama & Silverman, 1986; Thornton & Gilden, 2007) and reading (e.g., Engbert, Nuthmann, Richter, & Kliegl, 2005; Reichle, Rayner, & Pollatsek, 2003). However, this debate is mainly centered upon attention to objects and words, and hence on the input side of processing, rather than upon attention to response control on the output side. Since the present study specifically addresses the issue of multiple response control, it seems important to take a further look at our present data to speculate about the mode of processing (serial vs. parallel) in the CIP based on the relevant theoretical frameworks that model attention in multiple action control. 

The idea of serial response selection mechanisms in multiple action control was mainly derived from PRP studies in which processing overlap is varied by manipulating the time interval between the onset of two stimuli (Welford, 1952). It was further developed by assuming a central response selection bottleneck (RSB, Pashler, 1994). A clear prediction of the RSB framework is that whenever two responses are processed in close temporal proximity, overall RTs increase and especially the second of the two responses should suffer from RT costs, since response selection for the second response has to wait until selection of the first response is completed. A corresponding pattern of results has been replicated repeatedly within a range of PRP studies (Pashler, 1994), with eye movements to salient peripheral stimuli representing a single but notable exception (Pashler et al., 1993). 

Some of our present results appear to be in line with the assumption of serial response selection in the CIP. For example, Experiment 1 showed that the slower (vocal) response modality exhibits greater dual-response costs than the faster (saccade) response modality, a finding that is similar to the typical PRP effect in the form of a particular prolongation of the second response latency due to the first-come-first-served principle of a processing bottleneck. Although the overall effect size in Experiment 1 is much smaller than a typical PRP effect, it has been claimed that adverse effects of a central bottleneck can be comparatively small under certain conditions (e.g., Anderson, Taatgen, & Byrne, 2005). Nevertheless, the data from Experiment 2 are not in line with the assumption of smaller costs for the faster response modality, since the data clearly indicate equal (or even relatively greater) costs for the faster (manual) response. Furthermore, one general prediction of the PRP framework, namely greater overall conflict for conditions in which two responses are processed in closer temporal proximity, is not compatible with our data, since overall dual-response costs in RTs were not affected by temporal response distance. Thus, and in line with similar data from a previous CIP study (Huestegge & Koch, 2010), our results do not appear to fit into a classic bottleneck framework. 

However, it should be noted that our present study design, the CIP, substantially differs from the PRP paradigm, which might prevent a meaningful application of the RSB framework to our present data. Most importantly, our design involved two responses based on a single stimulus, which has two potential implications. First, it is possible that only a single compound response selection occurs for both responses, effectively eliminating the need for serial or parallel response selection processes. Evidence for this claim has previously been presented by Fagot and Pashler (1992), who showed that compound selection occurs when two responses are mapped to the same (attribute of a) stimulus. However, they only examined conditions with compatible responses, so it is unclear to what extent this reasoning should also hold for incompatible response demands. Thus, a second possibility is the assumption of two distinct response selection processes in the CIP. Based on traditional serial processing stage logic, response selection should begin right after the completion of stimulus processing, i.e., both response selection processes should begin with an equal probability for both responses and should thus (on average) lead to similar costs. However, the prediction of similar costs is clearly at odds with the results from Experiment 1 and the proportional cost data in Experiment 2. Finally, under the assumption of two parallel response selection processes, one might have expected that the shorter response would always show fewer dual-response costs, because a short response has less time to suffer from negative interference, or that both responses would show an equal amount of costs based on the time period in which both responses overlap. However, these general, more rigid mechanisms are clearly at odds with the present results that demonstrate a priority of (slower) vocal responses over (faster) manual responses. Again, the data rather point toward a highly flexible resource scheduling regime. Taken together, the notion of simultaneous processing of both responses appears to be a better overall framework to explain our present data compared to the concept of a serial response processing bottleneck, which may be more suited to account for typical PRP experiments and data. 

Another concern for the interpretation of our present data could be the claim that our response distance manipulation was not effective enough to invoke much conflict. Within the PRP framework, a similar discussion is based on the notion of a latent response selection bottleneck (i.e., when the duration of response selection is too short to allow for the occurrence of a processing slack in a reasonable number of trials; see Ruthruff, Johnston, Van Selst, Whitsell, & Remington, 2003). Indeed, the single-RT differences between response modalities were still substantial (about 200–300 ms) in both experiments. However, at least two observations speak against a lack of effectiveness of our response distance manipulation. First, the voice key procedure implemented here is known to overestimate actual speech onset times. In fact, we ran a control experiment with a similar setup and the same vocal responses and found that the voice key overestimated actual speech onset times by about 230 ms. If we take this into account, the actual response onset times (in single-response conditions) are much more similar (and thus, the effectiveness of our response distance manipulation is more effective) than suggested by our voice key data. Second, and probably more importantly, the elevated error rates in the close-responses condition of Experiment 1 clearly showed that cross-response conflict increased. Additionally, the observed trade-off of dual-response costs across modalities as a function of response processing distance also suggests conflict resolution processes. These empirical markers of conflict clearly do not support the assumption that our manipulation of response distance was ineffective. 

While the results from the previous CIP study (Huestegge & Koch, 2010) also suggested parallel instead of serial processing, there were also important differences: Dual-response costs in that study were unaffected not only by response processing distance overall but also individually for each response modality. Based on these previous results alone, one might have concluded that parallel processing occurs but that the two processing streams do not interact (i.e., independent parallel processing). In contrast, the present results rather suggest a strong and flexible interaction between response modalities (dependent interactive parallel processing), which is especially apparent in the observation that reduced temporal separation of response processing increased error rates in Experiment 1 (i.e., response processing effectiveness in one response modality was affected by the relative temporal distance of the context response). This interaction between response modalities is further underlined by the fact that the temporal structure of responses (and the specific combination of response modalities across experiments) affected the pattern of resource allocation across modalities. 

Because our observation that eye movements are preferred over other effector systems is a very recent finding and the literature on oculomotor action in multitasking is currently quite sparse, the origins and specific mechanisms of oculomotor dominance still need to be examined more closely. Although the present data present clear evidence for effector system prioritization, there seems to be room for flexible adjustments based on the particular task demands (e.g., resolving response conflict). Hence, it appears principally possible that the ordinal structure of effector system prioritization may change under certain task conditions, so that, e.g., eye movements would not be processed with the highest priority anymore. The finding that multitasking performance also depends on the specific combinations of input (i.e., stimulus) modality and output (i.e., response) modalities (Hazeltine et al., 2006; Stephan, Koch, Hendler, & Huestegge, 2013), in that some S-R combinations are processed with less interference than others, might serve as a starting point for further investigations regarding the flexibility of effector system prioritization. 

Conclusions

The present results (with those from Huestegge & Koch, 2010) suggest that multiple response processing across response domains occurs in parallel, but in a strongly interactive and flexible manner. The interaction mechanisms are based on constraints imposed through the temporal structure and modalities of the response demands involved, with a priority on the control of the oculomotor system. The evidence for interactive processing across response processing streams also fits into a larger framework suggesting that information crosstalk is a major factor determining the efficiency of multiple response control (Navon & Miller, 1987; see also Bekkering et al., 1994; Logan & Gordon, 2001). 

We would like to conclude that despite some limitations (e.g., the limited range for manipulating response processing distance), the CIP appears to be a useful and powerful paradigm to study multiple response control in order to move theory on multiple action processing forward by overcoming some of the inherent drawbacks of other paradigms, in particular the necessity of serial stimulus presentation in the PRP paradigm (see Meyer & Kieras, 1997). Thus, the CIP nicely complements other paradigms in the field of multiple response control, such as task switching (Stephan et al., 2013; see Kiesel et al., 2010, for a review), the PRP paradigm, and other dual-task paradigms. 

Finally, and with respect to the cognitive processes underlying oculomotor control, our results also show that it is theoretically rewarding to view the eyes not only in terms of an input modality (i.e., as a prerequisite of visual information uptake), but also as a response modality on their own that dominates other, concurrent response demands in other effector domains (oculomotor dominance effect), but in an interactive way. 

Acknowledgments

The authors thank Janika Thielecke for the collection of the data, and those who participated in the study. The present research was funded by the German Research Council (DFG HU 1847/3-1) and the University of Würzburg (funding program Open Access Publishing). 

Commercial relationships: none. 

Corresponding author: Aleksandra Pieczykolan. 

Email: [email protected]. 

Address: Institute of Psychology, University of Würzburg, Würzburg, Germany. 

Footnotes

¹  This method of measurement overestimates vocal response times by about 200 ms (as determined through different, but comparable experiments using off-line, manual speech-onset coding) because the sound pressure level at the beginning of a spoken word is significantly lower compared to the average sound pressure level of the word. Thus the voice key trigger cannot detect the immediate speech onset.

Footnotes

²  Note that this factor is equivalent to the variable response distance in Experiments 1 and 2. This different labeling was utilized to facilitate the comprehension of the argumentation.