We show that a Markov model captures the variance in eye and hand movement sequences in a natural task such as making a sandwich. Observing the different ways subjects perform the task allows the automatic decomposition into subtasks. The different ways of doing the task can then be described as alternate possible sequences of primitive operations, including eye movements and hand movements. Each such sequence fully characterizes a sandwich-making behavior. The transitional probabilities between subtasks are then computed from human data. The resultant model can produce new variations, which can be executed by a graphical human model in virtual reality with eye movements, body movements and object manipulation capability.
The model can explain almost all eye fixations observed in the course of sandwich-making, including anticipatory fixations to objects that are to be manipulated in the future. We explain such anticipatory fixations as initiated to update visual memory of objects relevant to future subtasks. The memory update facilitates upcoming visual search and visual guidance of hand movements. In this model, memory uncertainty initiates look-aheads probabilistically with other task specific parameters.
Experiments with human subjects making sandwiches show that the Markov model of subtask planning fits human data almost exactly. The mean number of anticipatory fixations in 10 trials averaged over 3 subjects was 3.10 with a s of 1.13 whereas the mean for the computer simulation was 3.09 with a s of 1.08. Thus anticipatory fixations can be seen as given advance notice of a visuo-motor plan.
This work was supported by NIH grants EY05729 and RR09283.