The next step was to calculate the successor representation (SR; Dayan,
1993) for each scanpath. We used a temporal-difference learning algorithm to extract long-range statistical regularities from the sequence. The algorithm treats each scanpath as a first-order Markov chain with the 10 AOIs comprising a discrete, finite state space (Dayan,
1993; White,
1995). The algorithm is incremental and builds a 10 × 10 SR matrix
M . The matrix is initialized with zeros and then updated for each transition in the sequence. Consider a transition from state
i to state
j. The
ith column of the matrix—the column corresponding to the “sender” AOI—is updated according to
where
I is the identity matrix, each subscript picks a column in a matrix,
α is a learning rate parameter (0 <
α < 1), and
γ is a temporal discount factor (0 <
γ < 1). In words, upon observing a transition
i→
j, the set of expected successors (
M i ) for the sender
i is updated to include the receiver
j (represented as a unit column vector
I j ) and the predicted set of successors (
M j ) for the new location
j, discounted by
γ. The latter term is the key to extending the event horizon to encompass both immediate and long-range transitions—it includes the discounted future states in the prediction from the current state. For example, suppose a participant scans the top row of a Raven problem systematically from left to right: 1→2→3→1→2…. Then, the successors of location 1 will include
both location 2 and, weighted by
γ, location 3. By contrast, a first-order transition matrix would include only the association between 1 and 2. After traversing the whole scanpath, the estimated SR matrix approximates the ideal SR matrix, which contains the temporally discounted number of expected future fixations on all AOIs (rows), given the participant just fixated on any individual AOI (column). Note that the entries in the SR matrix are not probabilities. They are (discounted, expected)
numbers of visits, and thus, the sum across each column of the ideal SR matrix equals
1 provides additional technical details.