Prediction errors in surface segmentation are reflected in the visual mismatch negativity, independently of task and surface features

Abstract

The visual system quickly registers perceptual regularities in the environment and responds to violations in these patterns. Errors of perceptual prediction are associated with electrocortical modulation, including the visual mismatch negativity (vMMN) and P2 event-related potential. One relatively unexplored question is whether these prediction error signals can encode higher-level properties such as surface segmentation, or whether they are limited to lower-level perceptual features. Using a roving standard paradigm, a triangle surface appeared either behind (featuring amodal contours) or in front of (featuring real contours) a second surface with hole-like windows. A surface layout appeared for two to five repetitions before switching to the other “deviant” layout; lighting and orientation of stimuli varied across presentations while remaining isoluminant. Observers responded when they detected a rare “pinched” triangle, which occasionally appeared. Cortical activity—reflected in mismatch responses affecting the P2–N2 and P300 amplitudes—was sensitive to a change in stimulus layout, when surfaces shifted position in depth, following several repetitions. Specifically, layout deviants led to a more negative P2–N2 complex at posterior electrodes, and greater P300 positivity at central sites. Independently of these signals of a deviant surface layout, further modulations of the P2 encoded differences between layouts and detection of the rare target stimulus. Comparison of the effect of preceding layout repetitions on this prediction error signal suggests that it is all or none and not graded with respect to the number of previous repetitions. We show that within the visual domain, unnoticed and task-irrelevant changes in visual surface segmentation leads to observable electrophysiological signals of prediction error that are dissociable from stimulus-specific encoding and lower-level perceptual processing.

Introduction

The predictive coding view of the brain suggests that we are constantly building models of our environment, and that the ongoing interaction between the predictions of these models and our sensations drive perception (Friston, 2005). At all stages of processing, sensory information is compared against predictions of expected sensory events made by higher-level perceptual areas, and the differences between these—prediction errors—are propagated upwards to update perceptual models of the environment (Rao & Ballard, 1999). 

The mismatch negativity (MMN) is well known as an electrophysiological signal of sudden changes in regularities in the auditory stream (Näätänen, Kujala, & Winkler, 2011). For example, if multiple tones of a specific frequency (standards) are followed by a tone of a deviant frequency (an oddball or deviant), the latter is accompanied by a negative voltage wave, even when other stimulus properties are controlled for. This effect is best elicited when observers' attention is engaged elsewhere, suggesting an automatic process (Näätänen, 1990). A growing body of literature describes a visual equivalent to this signal: the visual mismatch negativity (vMMN) (Tales, Newton, Troscianko, & Butler, 1999). Using the oddball task and related paradigms, deviancies from established regularities in visual features trigger negative-going voltage waves at varying latencies, when compared against standard trials and other control conditions (Astikainen, Lillstrang, & Ruusuvirta, 2008; Kimura, Katayama, Ohira, & Schröger, 2009). 

Mismatch responses in the visual hierarchy

The vMMN has been used to explore visual processes that track regularities in sensory information. Commonly, these have included lower-level visual features such as oriented lines and edges, color, and motion (for reviews, see Pazo-Alvarez, Cadaveira, & Amenedo, 2003; Kimura, 2012; Stefanics, Kremláček, & Czigler, 2014). For these elementary visual features in particular, there is an ongoing debate on whether mismatch effects might be better explained by neural refractoriness or stimulus-specific adaptation (SSA; e.g., Czigler, 2007; O'Shea, 2015; Stefanics, Kremláček, & Czigler, 2016), rather than a genuine memory-based or predictive process. Higher-level features, in contrast, may be less susceptible to SSA by design, as the lower-level sensory inputs comprising a higher-level categorical representation can vary without forming regularities (Winkler & Czigler, 2012). 

There is growing evidence that the visual system forms expectancies of complex, high-level properties of visual stimuli. One example is in visual language processing, where findings suggest preattentive processing of lexical, phonological, and speech gestural information, even in the absence of auditory information. Shtyrov, Goryainova, Tugin, Ossadtchi, and Shestakova (2013) found mismatch signals when task-irrelevant stimuli changed from words to pseudowords, and vice versa. In a stream of Chinese homophonic characters, participants automatically detected violations of expected phonological tone (Wang, Liu, Wu, & Wang, 2013). Studies have shown mixed evidence for deviance detection in expected speech mouth movements (Saint-Amour, De Sanctis, Molholm, Ritter, & Foxe, 2007; Ponton, Bernstein, & Auer, 2009; Files, Auer, & Bernstein, 2013), perhaps due to the complexity in presenting these dynamic stimuli. 

Visual mismatch signals have also been studied in the context of face perception. An increasing number of studies now describe a mismatch signal for face emotional expressions (Susac, Ilmoniemi, Pihko, & Supek, 2004; Zhao & Li, 2006; Stefanics, Csukly, Komlósi, Czobor, & Czigler, 2012; Astikainen, Cong, Ristaniemi, & Hietanen, 2013; Vogel, Shen, & Neuhaus, 2015; Kovarski et al., 2017). Similarly, face gender regularities have also be shown to be tracked by the visual system (Kecskés-Kovács, Sulykos, & Czigler, 2013; Wang et al., 2016). The tracking of these features requires that accompanying deviance in the lower-level perceptual inputs comprising face stimuli be discounted, even though these inputs are also contributing to higher-level holistic representations (Hayward, Crookes, Chu, Favelle, & Rhodes, 2016). 

Considering that both visual language and face processing show evidence of high-level perceptual predictions, it stands to reason that a similar aspect of visual processing with important survival value—object perception—should show an analogous sensitivity to regularities (Kersten, Mamassian, & Yuille, 2004). In the auditory modality, there is a wealth of evidence that expectancies are formed for auditory “object” representations, independent of lower-level features (Sussman, 2004; Winkler & Czigler, 2012). Yet, in the visual mismatch literature, only a few studies have explored object processing and perception per se. 

An early study in the mismatch literature explored changes to the form of a stimulus. Czigler and Csibra (1990) presented observers with a rapid serial visual presentation (RSVP) stream in which stimuli were comprised of angles forming a diamond (<>). On 10% of trials, a deviant angle arrangement appeared, forming a cross (><). Deviants relative to standards elicited large negative deflections between 100 and 300 ms after stimulus onset. This early investigation was limited in that participants' attention was directed toward the changes in stimulus form, and stimulus forms were not counterbalanced as standard and deviant. Thus, although the stimuli were arrangements of oriented lines, their perception arguably could involve more complex object processing. 

More recently, researchers conducted an investigation of how mismatch effects would be driven by changes in irrelevant object relations (Müller, Widmann, & Schröger, 2013). There, participants saw a stream in which each presentation featured two ellipsoid “objects” and two separate targets that were to be compared. Targets appeared either within a single ellipse or across two ellipses; on deviant trials (12.5%), the target-to-object association changed. On these deviant trials, participants were slower to respond to targets, and showed a posterior-temporal negativity that overlapped the P2 and N2 components. Localization implicated the inferior temporal gyrus as the source of this mismatch signal. This experiment followed up on an earlier mismatch study that suggested that object processing occurred and drove a mismatch response despite the absence of attention to object stimuli (Müller et al., 2010). 

Mismatch responses in surface segmentation

An aspect of visual object perception that has not been investigated in regularities is surface segmentation. When distal surfaces and edges partially occlude one another, the visual system is able to infer border ownership of the common objects to which these belong, through good continuation (Kellman & Shipley, 1991), depth cues (Nakayama & Shimojo, 1992), or global percept patterns (van Lier, van der Helm, & Leeuwenberg, 1994). Whether dominated by local features (i.e., edge T-junctions) or internal volume representations, surface perception is clearly a deeply hierarchical process involving multiple bottom-up inputs and contextual cues from above (Kersten et al., 2004; Tse, 2017). The identification of object surfaces and their localization in space also plays a central role in hierarchical vision, providing feedback for lower-level visual inferences (Kok & de Lange, 2014; Wokke, Vandenbroucke, Scholte, & Lamme, 2013) and interacting with scene perception to allow human action with the environment (Oliva & Torralba, 2007). 

Current theories of predictive coding in perception emphasize that sensory information at multiple perceptual levels should be integrated in a Bayesian hierarchical fashion (Rao & Ballard, 1999; Wacongne et al., 2011). As visual surface segmentation plays a pivotal role in the visual hierarchy, it is worth considering whether it behaves in a predictive coding manner. We ask whether surface perception shows a trademark sign of ongoing inference of environmental regularities, specifically, the visual mismatch negativity. 

In this study, we investigate ERP (event-related potential) mismatch signals in the context of surface segmentation. To do so, we compare electrophysiological responses to perceptually identical stimuli which either repeat the preceding surface layout or deviate from it. These stimulus layouts differ in terms of where visual surfaces are positioned in depth, relative to one another. Depth cues are given based on partial occlusion of a background surface by a foreground surface. 

In addition, we investigate whether mismatch signals are affected by the strength of prior beliefs. We measure prior evidence using the number of prior presentations of a stimulus layout and quantify how this modulates effects of layout deviation. Importantly, we seek to demonstrate whether these signals occur independently of stimulus perceptual properties and task demand. 

We hypothesized that changes in surface layout across trials would trigger an electrophysiological mismatch response, and expected that this mismatch response would be graded with respect to previous repetitions. Although we had no specific expectations about responses to stimulus layouts themselves, any effect of stimulus precepts should not interact with (and potentially explain) any observed mismatch response. 

Method

We employed a roving standard paradigm, a variation of the oddball paradigm, in which repeated stimulus features can form regularities before being changed in an oddball-like event. Yet, across an experiment, no stimulus features are more common than others, and only the immediately preceding stimuli and their features determine which stimulus is a deviant. Thus, global stimulus frequency (Kenemans, Jong, & Verbaten, 2003) and stimulus perceptual features are excluded as potential explanations for any mismatch effect. 

Participants

We recruited 29 participants (15 female; mean age 23.9 years) for this study with an aim to analyze 25 participants, in line with similar electroencephalogram (EEG) research. Participants were recruited primarily from the student and staff body at the University of Auckland, New Zealand. All participants provided signed consent and were thanked with a $20 NZD voucher. All participants reported normal or corrected vision and no sensitivity to flickering stimulation or history of migraines. The experiment was conducted in accordance with the principles embodied in the Declaration of Helsinki and approved by the University of Auckland Human Participation Ethics Committee, reference number 013154. 

Data from three participants were excluded immediately after data collection for the following reasons: dozing during the experiment, data recording error, and voluntary withdrawal. During EEG data analysis but prior to condition unblinding, one further participant was rejected due to significant EEG signal artefacts. The recruitment and rejection process followed criteria laid out during study preregistration. 

Stimuli

On each presentation, participants saw two visual surfaces and a light source (Figure 1). The first surface was a black Plane or surface spanning the entire screen perforated by three “holes” (each 2.4° in diameter) arrayed in an equilateral triangle, with each hole positioned at 3.4° eccentricity from display center. Through the holes, a 50% gray background (20.5 cd/m²) was partially visible. A second surface, an equilateral Triangle circumscribing a radius of 3.8° (i.e., 6.6° on a side) was arranged so that the vertices were approximately aligned with the holes of the Plane. The Plane and Triangle were always centered on the display, but rotated together 1° about the screen center between presentations, so that sensory adaptation to oriented edges did not occur. 

Figure 1

 

Top left: Examples of each of four possible stimulus configurations; orientation varied across trials, and color and parameters of the illuminant varied randomly. Top right: State diagram for stimulus selection across presentations, using a Plane-Front Plane-Lit configuration (in magenta) as an example starting state. A repeat of Lit Surface never occurred, while a Surface Layout Repeat and Deviant were equally likely. Transitions were also constrained by the minimum and maximum number of Layout Repeats (not illustrated). Bottom: Example RSVP stream; layout Deviants are bordered in red. The factor Prior Evidence, the trial-level coding of preceding repeats, is shown in brackets. See also Movie S1.

To promote the perception of two separate surfaces, a procedurally-generated Light source appeared cast upon either the Plane or Triangle, revealing distinct surface boundaries differing in depth. The Light was an ellipsoidal Gaussian consisting of a random offset from display center (up to 2.1°), orientation, sigma (2.3-3.7°), and aspect ratio. The Light could take one of five isoluminant colors (red, green, blue, orange, magenta, or teal; central luminance: 15.0–26.2 cd/m²). To prevent color adaptation and the detection of regularities, color was randomly selected with the constraint that the same color could not appear twice in three consecutive presentations. Unless lit by the Light source, the Plane and Triangle surfaces were black (0.9 cd/m²). 

These visual objects were presented synchronously to participants in a rapid serial visual presentation (RSVP) paradigm at approximately 0.87 Hz. On each presentation, the Light and holes in the Plane appeared at full luminance, allowing the surface edges and configuration to be perceived. Luminance of the entire stimulus decayed by approximately 5% every 17 ms, reducing to zero luminance after 600 ms. Following offset, there was a randomized interstimulus interval (ISI) of 300 to 600 ms before the following presentation. 

Arrangement in depth of the two surfaces and light source was manipulated via two factors to create four stimulus configurations. 

Surface layout

The Triangle appeared either in front of or behind the Plane in depth. The Plane-Front layout suggested a triangle behind three holes in the plane, i.e., an amodal Kanizsa triangle (Kanizsa, 1987). The Triangle-Front layout showed a triangle above or occluding three holes in the plane, with triangle contours mostly visible. Potentially, layouts could be perceived in other ways, such as circles appearing as objects rather than as holes. Surface Layout was the primary property of the stimuli that we controlled for regularity in presentations. We treated it as a global perceptual feature for the purposes of inferring perceptual prediction errors. 

Lit surface

Orthogonally to the Surface Layout, Lit Surface varied based on which surface was illuminated by the Light. That is, in Triangle-Lit configurations either the entire triangle was visibly lit (in the foreground) or only corners were visible (through holes in the Plane). In Plane-Lit configurations, either the entire Gaussian patch was visible with a black triangle in background, or was obscured by a black triangle. The interaction of this factor with Surface Layout can be considered as the apparent position in depth of the Gaussian light source relative to the surfaces: the Light could appear closer to the viewer, illuminating the foreground figure, or sandwiched between the two figures, illuminating the background figure. The latter interpretation notably also coincides with whether the configuration was illuminated at the fovea. The Lit Surface alternated on every stimulus presentation, and therefore was not investigated in predictive coding, but only as a stimulus factor. 

Other than the aforementioned factors, configurations were collapsed across other randomized stimulus factors such as light source color and stimulus rotation. 

The main factor of interest was repetitions and changes to Surface Layout across stimulus presentations. The Surface Layout of a given stimulus configuration was generally equiprobable (p = 0.5) and randomly selected on-line. Because each Surface Layout was equiprobable on a presentation, a repeat or deviant of the preceding layout was also equally likely. Thus, feature deviance was not a rare occurrence, unlike in the usual roving standard design in which transition probabilities change across a microsequence (e.g., Stefanics, Heinzle, Horváth, & Stephan, 2018). An exception to the equiprobable presentation was that a minimum and maximum number of layout repetitions was enforced within a microsequence, such that a given Surface Layout would always appear at least twice, and never more than five times consecutively (i.e., 2–5 within-layout repeats). The Lit Surface alternated on every presentation. A state diagram illustrating the selection of stimulus factors across the presentation stream is shown in Figure 1. 

Example stimulus streams are presented in Supplementary Movie S1 and illustrated in Figure 1. 

Procedure

Participants sat in a darkened Faraday cage, 57 cm from a 24-in. LED monitor (SyncMaster; Samsung, Suwon, South Korea S24B350H), which displayed the experiment at 60 Hz. The experimental procedure was coded in and presented using MATLAB 2012b (MathWorks, 2012) and PsychToolbox (Kleiner, Brainard, & Pelli, 2007). 

Participants were asked to fixate on the center of the screen and attend to the stimuli as they appeared. Simultaneously, they performed a rare stimulus detection task: occasionally, a “pinched” or “scalloped” triangle appeared in the array. These targets appeared on every 5–46 presentations following a Gamma distribution, with a peak of 19 trials between targets. Participants reported the presence of this target by pressing the spacebar before the following stimulus presentation; participants were informed that this task was unspeeded and that they should focus on responding accurately. Before beginning the experiment, participants were familiarized with the stimuli and asked to point out the pinched triangles in a set of examples. Fixation was not explicitly controlled or tracked, although participants were advised that attending to the center of the display would improve performance. Participants were provided error feedback immediately after detection misses and false alarms. There were 2,500 stimulus presentations in the experiment, broken up into 150 trial blocks, each lasting 3 min. In addition to breaks following each block, two longer breaks occurred after 20 and 40 min to check EEG net impedance and re-wet sensors when necessary. 

Electrophysiology

Simultaneous to the experimental procedure, electrical activity was recorded from the scalp via a 128-electrode Geodesic Sensor Net running on a NetAmps 300-ampere amplifier (Electrical Geodesics Inc., Eugene, OR). Scalp activity was recorded at 1000 Hz and with respect to the vertex electrode. Prior to the experimental procedure, all sensors were adjusted to ensure an impedance below 40 kΩ, and this impedance was checked again during longer breaks 20 and 40 min into the experiment. 

After data collection, electrophysiological data for each participant were processed for analysis following a preregistered procedure, outlined here. This procedure was carried out primarily using EEGLAB version 14.1.1 (Delorme & Makeig, 2004) in MATLAB 2017a. First, 20 face and neck channels were removed from the data set and not further analyzed, leaving a 108-electrode montage. All EEG data were then downsampled to 250 Hz. This continuous data were then cleaned using the PREP Pipeline (Bigdely-Shamlo, Mullen, Kothe, Su, & Robbins, 2015), a standardized preprocessing toolbox, which flags bad electrodes for rejection and calculates a robust average reference. All channel data were then bandpass filtered between 0.5 Hz and 50 Hz using a Hamming windowed sinc zero-phase finite impulse response (FIR) filter (order = 1,650, beta = 5, stopband attenuation = −53 dB, max. passband deviation = 0.22%, transition band width = 0.5 Hz) (Widmann, Schröger, & Maess, 2015). 

Each participant's EEG data were then cleaned via a multi-step artefact rejection procedure. First, continuous data were visually inspected, and timespans were manually rejected, which showed clear paroxysmal activity or other abnormal spatially-distributed artefacts. Time periods with gross muscle activity or multiple blinks/eye movements were also removed. One participant's data were rejected at this point due to unexplained high-variance activity, which rendered the EEG signal irrecoverable. Next, a principal component analysis was performed using the runica implementation in EEGLAB to isolate the 35 principal components in each data set (while excluding channels previously flagged). Of these components, those representing clear artefacts with high explained variance were manually identified following guidelines in EEGLAB (Delorme & Makeig, 2004) and the ICLabel online tutorial (Pion-Tonachini, Kreutz-Delgado, & Makeig, 2019). These primarily included eye blinks and movements, heartbeat and other stereotypic muscle activity, and electrode malfunction over large timescales. Flagged component activities were subtracted from the data. Abnormal channels identified earlier using PREP Pipeline were then spherical-spline interpolated. 

The continuous data were then divided into up to 2,500 discrete epochs lasting 1,000 ms, from 200 ms before stimulus onset. Voltages were then baseline corrected by subtracting the mean pre-onset voltage in the baseline period (-200 ms to 0 ms) from each electrode in each epoch. 

As a final step we performed epoch-wise channel interpolation to reduce the effect of spatially- and temporally-isolated abnormal activity. This procedure identified incidental electrode issues without the need to reject channels throughout the entire session, or reject entire epochs due to spatially limited artefacts; it is adapted from a method in the FASTER toolkit (single-channel, single-epoch artifacts; Nolan, Whelan, & Reilly, 2010). We separately calculated (i) the log of the variance of a channel relative to other channels within a given epoch (spatially-deviant activity), and (ii) the log of the variance of an epoch relative to other epochs for a given channel (temporally-deviant activity). Channels within epochs for which the sum of the Z-scores of these measures was above 4 were interpolated using a spherical-spline method. Via this procedure, 1%–3% of abnormal data for each participant was interpolated. Finally, epochs in which more than 20 of 108 channels were interpolated in the previous step—likely representing artefacts overlooked during visual inspection—were rejected from further analysis. 

Analysis

Conditions

Stimulus presentations were divided into conditions in several ways. As mentioned, each stimulus was coded as one of two Surface Layouts (Triangle-Front or Plane-Front) and one of two Lit Surfaces (Triangle-Lit or Plane-Lit), based on the perceptual properties of the stimulus. To investigate the effects of perceptual prediction and regularities in the RSVP stream, presentations were also categorized based on the stimuli that immediately preceded them in a microsequence. The factor of Layout Deviance (layout Deviant or layout Repeat) reflected whether a given presentation featured the same or different Surface Layout from the immediately preceding stimulus. A second factor, Prior Evidence, coded the number of layout repeats prior to a deviant or repeat presentation. Due to the roving standard design, Prior Evidence had three unbalanced levels, as the presentation order of stimuli followed a state diagram, which probabilistically determined the relative frequency of each level. A microsequence of two repetitions of a given Surface Layout occurred twice as often as three repetitions, which in turn occurred twice as often as four repetitions. 

As a constraint to stimulus selection in the RSVP stream, Surface Layouts always appeared 2–5 times consecutively in a microsequence. Thus, a Deviant was always followed by at least one repetition of the same Surface Layout (a First-Repeat), while five repetitions of a layout was always followed by a layout Deviant (a Fifth-Deviant). These latter conditions differed from other repeat and deviant trials in an important regard. Because they occurred obligatorily, they could be anticipated by participants; in other words, the likelihood of a given layout was not balanced with other conditions, which could affect neural responses to statistical expectancies in microsequences. Furthermore, the factor of Layout Deviance was not balanced for these situations, in that there were no “First-Deviant” or “Fifth-Repeat” trials. Accordingly, these conditions were excluded from both behavioural and physiological analyses. 

Some trials were excluded from calculations and analysis when it was considered that stable stimulus regularities were interrupted or not formed. Specifically, detection task hits, misses, and false alarms were excluded (except as noted as follows). All first presentations in a block, following a break, or following a task error were also excluded. 

Following data preprocessing and epoch rejection, an average of 2,467 of 2,500 presentations (range: 2,307–2,496) remained for analysis. ERPs of interest were calculated using an average of 661 Layout Repeat trials (range: 574–714) and 660 Layout Deviant trials (range: 616–698). These were further subdivided by factors Surface Layout and Lit Surface, and separately by Prior Evidence. The analyzed condition cells with the fewest trials were Fourth-Repeat (95 trials; range: 73–116) and Fourth-Deviant (92 trials; range 78–106). 

Confirmatory analysis

We preregistered two hypotheses of interest prior to data collection, based on previously collected pilot data (n = 4), which are not otherwise used in this study. As a main effect, we expected modulation of ERPs that differentiated between the Layout Deviance of a stimulus (Surface Layout Repeat or Deviant from preceding presentations). In addition, we wished to see if any mismatch difference between Repeat and Deviant stimuli would interact with the degree of Prior Evidence, coded as the number of layout repetitions on immediately preceding trials. 

Analysis of preliminary data highlighted two time intervals and regions of interest associated with a change in layout. These spatiotemporal windows of interest were preregistered as focal points of further analysis (see figures). The early window was from 200 to 250 ms following stimulus onset, at posterior electrodes in a band from PO3 through POz to PO4. Specifically, this region encompassed electrodes 66, 67, 71, 72, 76, 77, and 84 in the 128-electrode EGI HydroCel GSN system. This window of interest roughly corresponded with the spatial and temporal peak of the P2 component, as well as part of the N2 component. A second positive modulation was identified over a diffuse central and parietal area, with long-lasting effects peaking after 300 ms. Accordingly we defined this later window of interest as 300–400 ms after stimulus onset, across electrodes 7, 31, 54, 55, 79, 80, and 106, approximately the area surrounding Cz and CPz. While this region and time window appeared associated with the P300, and layout deviants showed more positivity relative to layout repeats, unsubtracted ERPs in this window showed negative voltages relative to the rest of the scalp. It should be noted that these windows of interest were determined a priori, and any spatial and temporal empirical effects could be expected to vary from these windows. 

Average ERP voltages were calculated across electrodes and time points for each window of interest (as already stated). These mean voltages were then submitted to a four-way repeated-measures analysis of variance (ANOVA) with factors Layout Deviance, Prior Evidence, Surface Layout, and Lit Surface. We tested for a main effect of Layout Deviance and any two-way interaction with that factor, plus a three-way interaction between Layout Deviance, Surface Layout, and Lit Surface. Further interactions between Prior Evidence and stimulus factors Surface Layout and Lit Surface were not considered. 

As specific hypotheses, we expected a larger negativity in prediction-violating stimuli compared to prediction-confirming stimuli in the early window. We also expected the later window to show a positive difference in prediction-violating stimuli compared to prediction-confirming stimuli. Thus, we hypothesized a main effect of Layout Deviance for each window of interest. Further planned comparisons were performed for the effect of Prior Evidence on Layout Deviance, using a Tukey's HSD test for the difference in voltage means between Deviant and Repeat stimuli as predicted by Prior Evidence. 

As the factor of Prior Evidence was unbalanced in number of trials in the ratio 4:2:1, the number of epochs available to compute voltage measures varied, potentially affecting their signal-to-noise ratios. A common strategy in this scenario is to randomly subsample from conditions with more trials (Cohen, 2014). Here, we instead used all available epochs for each condition, even if epoch counts were imbalanced. For confirmatory hypotheses, parallel analyses were performed using systematic sampling for epochs. For each level of the condition of Prior Evidence, epochs were sampled with a sampling interval of 1, 2, or 4, as appropriate, to select approximately 100 epochs per level. The results of these analyses did not noticeably differ from the nonsubsampled results described as follows. 

Exploratory analysis

We performed several other analyses to characterize the electrophysiological correlates and behavioural effects of prediction violation in this task. These analyses were not preregistered and must be considered exploratory. 

Topography

We further explored how the spatial distribution of any mismatch signal might vary from the preregistered regions of interest. We calculated the voltage means for layout Deviant and Repeat trials, collapsed across other conditions, at each of 108 electrodes. Voltage differences for each electrode were averaged from time points in the two windows of interest, 200–250 ms and 300–400 ms. A two-way repeated-measures ANOVA with factors Layout Deviance and Electrode was performed and p-values calculated using Tukey's HSD correction for multiple comparisons. Figure 4 shows these spatial distributions for the two windows of interest. To illustrate changes across time in the spatial distribution of the Layout Deviance effect, similar ANOVAs (with corrections for multiple comparisons across electrodes) were performed for each epoch time point; these were not further corrected for multiple comparisons across time. The resulting spatiotemporal effects are shown in Supplementary Movie S2. 

Task behavior

As the target detection task was orthogonal to the experimental manipulation, we formed no hypotheses as to how task performance might vary across conditions. Nevertheless, there is good evidence in the mismatch signal literature that behavioral outcomes are sensitive to deviance in stimulus regularities. We therefore explored the effects of Layout Deviance, Surface Layout, and Lit Surface on detection responses. Because participants detected a pinched target with near perfect sensitivity, we focused on reaction times for correct detections. These were submitted to a linear mixed effects model (using the MATLAB function fitlme), which took the form:  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}{\rm{RT}}\sim 1 + {\rm{Layout\ Deviance}} \times {\rm{Surface\ Layout}} \times {\rm{Lit\ Surface}} + \left( {1 + {\rm{Layout\ Deviance}} \times {\rm{Surface\ Layout}} \times {\rm{Lit\ Surface}}\,|\,{\rm{Participant}}} \right)\end{equation}

 

This model specification accounts for the possibility that all factors could vary across participants. 

Task electrophysiology

Finally, we investigated how evoked responses to targets would differ from the more frequent non-targets. Pinched targets appeared on approximately one in twenty trials. A further point of interest was how any effect of target detection might interact with layout repetition or deviance; layout deviant trials themselves were less frequent than layout repeat trials. As a result of these imbalances, the trial mass contributing to each ERP average was severely reduced. Therefore, we instead used a linear mixed effects regression to model ERP responses in both regions of interest at specific time intervals. Unlike repeated-measures ANOVAs, mixed effects regression models are better suited to designs where the number of observations greatly varies across conditions (Gelman & Hill, 2007). 

Specifically, the factor of Target Presence could affect ERPs in windows of interest, and the presence of a target could also interact with deviance in a pattern of layout presentations. To describe this relationship, we specified the following model:  

\begin{equation}{\rm{Voltage}}\sim 1 + {\rm{Target\ Presence}} \times {\rm{Layout\ Deviance}} + \left( {1 + {\rm{Target\ Presence}} \times {\rm{Layout\ Deviance}}\,|\,{\rm{Participant}}} \right)\end{equation}

 

This model allows that the effect of manipulated factors on mean voltages in times and regions of interest could vary across participants. 

Results

Behavior

As expected, participants found the behavioural task—the detection of a pinched triangle—to be trivial. The grand mean hit rate and correct rejection rate were both above 99%. All participants met the preregistered 90% hit rate criterion for inclusion. As detection rates were at ceiling, exploration of manipulation effects on behavior was limited to reaction times. 

Linear regression analysis of stimulus factors on reactions times showed an effect of Surface Layout: responses to pinched triangles when behind the plane were slower than when the triangle was in the foreground by 36 ms (p < 0.001, 95% CI: [23 ms, 48 ms]). As more of the triangle contour was visible in the latter condition, it is not surprising that concavity of the contour was more quickly reported. A similar main effect of Lit Surface showed that responses were slower when the triangle was lit by 32 ms, relative to when the plane was lit (p < 0.001, 95% CI: [15 ms, 49 ms]). Critically, there was no main effect of Layout Deviance, meaning that reaction times to target triangles did not differ between layout Deviants and layout Repeats (p = 0.29). There were no significant interactions between any of these factors. 

Electrophysiology

Early window

Planned analysis was performed on the first window of interest identified in pilot experimentation (Figure 2). A four-way repeated-measures ANOVA of mean voltages revealed a main effect of Layout Deviance: layout deviants showed attenuated P2 and increased N2 components relative to layout repeats, F(1, 24) = 42.34, p < 0.001. An exploration of the scalp distribution of this mismatch signal showed significant differences across the whole scalp, including a fronto-central positive component (Figure 4, left). Negative differences were strongest at lateral occipital electrodes, although more posterior than expected. 

Figure 2

 

Mean voltage ERPs in the posterior ROI for Surface Layout Repeats (blue) and Deviants (orange) at varying degrees of Prior Evidence (line patterning). Early window time of interest shown in yellow area. The grand mean effect of Layout Deviance (Layout Deviant minus Layout Repeat) shown in black, with gray ribbon illustrating the 95% CI of the difference. Averaged electrodes shown in topography (upper right).

As Mauchly's Test of Sphericity indicated that the assumption of sphericity across levels of Prior Evidence was violated, we corrected degrees of freedom using the Greenhouse-Geisser approximation (1959). Prior Evidence did not interact with Layout Deviance, F(1.43, 34.35) = 3.50, p = 0.056. Notably, when using decimated-epoch data, this interaction also failed to reach significance, F(1.79, 42.99) = 2.31, p = 0.11). Nevertheless, we followed up with a planned comparison of the effect of Layout Deviance across the values of Prior Evidence, amounting to a voltage subtraction of Deviant minus Repeat trials while controlling for the number of previous repetitions. Using Tukey's HSD correction for multiple comparisons, no pairwise comparison of the effect of Layout Deviance on presentations following two, three, or four layout repeats was significant (all ps > 0.1). 

The planned analysis of stimulus factors Surface Layout and Lit Surface was limited to their interaction with Layout Deviance, and the three-way interaction between Layout Deviance, Surface Layout, and Lit Surface. For the early window, none of these interactions were significant at the 0.05 level. 

Although unplanned, we looked also at whether the stimulus factors, Surface Layout and Lit Surface, could drive an early window modulation. The repeated-measures ANOVA also showed a main effect of Surface Layout: real contour configurations (Triangle-Front) showed a much more positive N1–P2–N2 complex relative to amodally-completed configurations (Plane-Front), F(1, 24) = 111.9, p < 0.001. This strong deflection was notable as early as the N1 and lasted several hundred milliseconds. Minor differences between levels of factor Lit Surface were not significant in this time interval (p = 0.36). 

Using a linear mixed-effects model, we also explored evoked responses to Target Presence (Figure 5). Trials with targets produced a weaker P2 by 1.00 microvolt (p < 0.001, 95% CI: [0.54, 1.45]). There was a main effect of Layout Deviance, confirming the repeated-measures ANOVA, with layout deviants showing a P2 reduced by 1.03 microvolts (p < 0.001, 95% CI: [0.74, 1.32]). These factors did not interact, p = 0.93. 

Late window

We expected the late window to reveal a difference wave associated with layout deviants, peaking after 250 ms. In a four-way repeated-measures ANOVA, a main effect of Layout Deviance was again noted, with layout repeats showing greater central negativity than layout deviant trials, F(1, 24) = 23.55, p < 0.001. Further visual inspection of the ERP response suggested that this modulation was long-lasting, spanning at least the 300–600 ms interval (Figure 3). The scalp distribution of the mismatch signal revealed that it encompassed a broad central and parietal region (Figure 4, right). 

Figure 3

 

Mean voltage ERPs in the central ROI for Surface Layout Repeats (blue) and Deviants (orange) at varying degrees of Prior Evidence (line patterning). Late window time of interest shown in yellow area. The grand mean effect of Layout Deviance (Layout Deviant minus Layout Repeat) shown in black, with gray ribbon illustrating the 95% CI of the difference. Averaged electrodes shown in topography (upper right).

Figure 4

 

Top: Scalp topography of the deviance effect in the early window (left) and late window (right). Bottom: Significant regions for the deviance effect, adjusted using Tukey's HSD correction for multiple comparisons. See also Movie S2.

Figure 5

 

Mean voltage ERPs in the posterior ROI for detected Layout Deviant Targets (solid green) and Layout Repeat Targets (dashed green). Equivalent ERPs shown for Nontarget Layout Repeats (blue) and Nontarget Layout Deviants (orange). Early window time of interest shown in yellow area. The grand mean effect of Target Presence (Target minus Nontarget) shown (black) with 95% CI of the difference (gray ribbon). Averaged electrodes shown in topography (upper right).

Again, Prior Evidence did not interact with Layout Deviance, F(2, 48) = 0.28, p = 0.76. Planned comparisons of the effect of Layout Deviance across the values of Prior Evidence showed no significant pairwise comparisons (all ps > 0.1). 

Stimulus factors Surface Layout and Lit Surface did not interact with Layout Deviance, save for the three-way interaction between Layout Deviance, Surface Layout, and Lit Surface. This three-way interaction was significant only in decimated-epoch data, F(1, 24) = 5.88, p = 0.023. This suggests that deviation from Surface Layout expectancies affected visual processing of certain stimulus configurations differentially. 

Unplanned analysis of the main effects of stimulus factors revealed no significant effect of Surface Layout (p = 0.83) in the late window, but a significant effect for Lit Surface. When the triangle surface was lit, central areas were more negative than when the plane was lit, F(1, 24) = 8.09, p = 0.009. 

Finally, we modeled the effect of Target Presence on voltages across central electrodes in the late window (Figure 6) with a linear mixed-effects model. There was a clear positive deflection of at least 1.28 microvolts associated with target detection in the window of interest (p < 0.001, 95% CI: [0.71, 1.86]). The main effect of Layout Deviance found in repeated-measures ANOVA was again seen (p < 0.001, 95% CI: [0.23, 0.47]). This deflection preceded a larger positive wave after 400 ms, possibly associated with a motor response of detection. There was no interaction of Layout Deviance and Target Presence, p = 0.23. 

Figure 6

 

Mean voltage ERPs in the central ROI for detected Layout Deviant Targets (solid green) and Layout Repeat Targets (dashed green). Equivalent ERPs shown for Nontarget Layout Repeats (blue) and Nontarget Layout Deviants (orange). Late window time of interest shown in yellow area. The grand mean effect of Target Presence (Target minus Nontarget) shown (black) with 95% CI of the difference (gray ribbon). Averaged electrodes shown in topography (upper right).

Discussion

In an RSVP roving standard task, we found that changes to established regularities in the arrangement of surfaces in depth—deviance in Surface Layout—produced unmistakable signals in two time ranges in scalp electrical activity. This deviance signal was separable from other strong signals encoding the Surface Layouts themselves, and encoding target detection on infrequent target present trials. Importantly, the absence of meaningful interactions between these latter signals and the deviance effect demonstrates that the deviance effect was indeed driven by across-trial pattern violation, rather than some other effect. Finally, we coded for the strength of pattern regularity at the trial-level, but found that this did not modulate the main deviance effect. This outcome suggests that although ERP correlates in the windows of interest reflected detection of a change in surface layout, they did not encode the degree of prior evidence as captured by the number of preceding layout repeats. 

Layout deviants affect the P2-N2 and P300 complexes

The clearest effect of Layout Deviance was observed in an early time window at posterior electrodes. Layout deviants were more negative than layout repeats, even when controlling for ERP effects driven by stimulus configurations. This effect emerged subsequent to the N1, peaking at the P2 but clearly overlapping the N2 component as well. We identify this negative difference as the visual mismatch negativity, although affected voltages were positive overall relative to a pre-stimulus baseline. 

Post-hoc investigation of the scalp topography of this early mismatch signal showed that the negative wave was strongest over posterior electrodes, specifically lateral occipital electrodes (Figure 4, left). Although we focused on the posterior negativity, a positive region was also observed across frontal and central regions, presumably reflecting the dipolar generator of the mismatch response. The posterior distribution accords with previous research on visual object processing, and amodal contour completion in particular. The lateral occipital complex is associated with object perception (Grill-Spector, Kourtzi, & Kanwisher, 2001), and feedback from this area has been shown to be functionally necessary for contour completion (Wokke et al., 2013). In the illusory Kanizsa figure, completed contours produce greater negativity in the N1–P2–N2 complex, and more so when those contours are amodally completed; the lateral occipital complex is again identified as a source (Murray, Foxe, Javitt, & Foxe, 2004; Ritzl et al., 2003). That the earlier mismatch signal was centered on known object-processing areas provides corroborating evidence that mismatch detection was specifically triggered by surface segmentation, a function closely tied to object and scene perception. 

The P2–N2 effect found here closely mirrors that seen in an earlier study of regularity violation in object-to-object relations (Müller et al., 2013). In that study, participants responded to a pair of targets that were situated within- or across-objects. Although this object relation was not relevant to the task, deviance in the relation of targets to objects led to a negative wave spanning the 200–300 ms interval, and centered on lateral occipital electrodes. 

A late “early” mismatch response

The mismatch response observed in our early window peaked after 200 ms. Earlier vMMN signals have been reported in the 100–200 ms range (Kovarski et al., 2017; Pazo-Alvarez et al., 2003), although these are not universal. In the current context, several causes could account for the relatively late emergence of the first signal of pattern deviance. 

First, careful stimulus control and removing sources of stimulus-specific adaptation (O'Shea, 2015; Stefanics et al., 2016) may have eliminated any earlier effects (Garrido, Kilner, Kiebel, & Friston, 2009). Adaptation to low-level perceptual features of stimuli was controlled for: light source properties were randomly generated for each presentation; color and edge contrast polarity changed on every trial; and orientation of the stimulus configuration varied in a systematic and consistent manner. Statistical regularities for these properties could not be established or were never violated; exogenous stimulus effects were expected to cancel out through destructive averaging or were accounted for in analysis. These simpler perceptual features could potentially drive regularity mismatch detection (perhaps through SSA) and have been previously investigated. For example, in previous work (File et al., 2017; Kimura et al., 2009), both early (100–200 ms) and late (200 ms onward) mismatch signals were found to oriented edges, but in control conditions where contributions to SSA were eliminated, only “genuine” mismatch signals from 200 ms after stimulus onset remained. Similarly, Kimura (2012) argues that modulation of the Visual N1 (150–200 ms) should be attributed to refractoriness and dissociated from a more “genuine” vMMN peaking after 200 ms. 

Deviances in higher-order stimulus properties may also explain the presence of later mismatch signals. Surface segmentation in this task likely occurs higher in the visual hierarchy, necessitating prior edge detection and perceptual completion. It would thus not be surprising if signals of layout deviance were delayed relative to mismatch signals to other visual properties. Potentially, identification of a “genuine” vMMN after 200 ms may require both the elimination of SSA-related activity and greater visual feature complexity. In a similar task involving object relations, Müller and colleagues (2013) also showed a long-lasting negativity emerging before 200 ms and peaking around 250 ms, but authors there also controlled for lower-level regularities such as stimulus orientation. Reduction of SSA components in Experiment 3 by File and colleagues (2017) eliminated the 100–200 ms vMMN signal; but, the complexity of stimulus visual features still modulated later mismatch differences. 

A late positivity associated with layout deviants

In a later window, a more diffuse but long-lasting positive difference was associated with layout deviants. While not exceedingly strong (around 0.5 microvolts), this signal was first notable at 300 ms after stimulus presentation, and lasted until at least 600 ms. Exploration of scalp topography showed that the central distribution encompassed frontal and parietal areas as well. 

When seen in previous literature, this late positivity is sometimes considered a part of the genuine mismatch response, although it has also been argued to be a result of attention processing resulting from detected sensory changes (Escera, Alho, Schröger, & Winkler, 2000; Kenemans et al., 2003; Stefanics et al., 2014). Indeed, the positive wave could represent an expression of the P3a or P3b, components associated with task demand and context-updating. Although there were no explicit task-related differences between layout deviant and layout repeat trials, attention to the triangle surface could nevertheless have contributed to these late components when features of the stimulus changed. 

Task demand effects were independent of deviance effect

Mismatch responses are known to occur automatically, without attention to the regularities that drive them (Pazo-Alvarez et al., 2003; Winkler, Schröger, & Cowan, 2001). To demonstrate this insensitivity to attention, tasks in mismatch procedures are commonly independent of the regularities of interest. In the current experiment, the task used, detection of concave surface contours, was orthogonal to the investigated regularity, surface depth in scene. Nevertheless, the task was incidentally related to the pattern: both discriminating features of a triangle and localizing that triangle in space likely relied on common perceptual processes. We explored ERP and behavioral responses to rare target triangles, although these were not the focus of the study. One question of interest was whether effects of target presence interacted with layout deviance. 

Target trials produced clear effects within and outside of the windows of interest, relative to non-targets (Figures 5 and 6). At posterior electrodes, a strong negative wave dissociated targets from nontargets, clearly notable in the P2 and N2 and beyond. More centrally, a positive deflection occurred from 300 ms, possibly representing a canonical P3a or P3b. This was followed by a further scalp-wide positive deflection—possibly associated with motor preparation and response, as it was absent in non-target trials. Critically, these effects of Target Presence did not interact with Layout Deviance in the windows of interest, and so were likely driven by separate perceptual processes. 

Behaviorally, there was no effect of Layout Deviance on hit rate or reaction times when detecting a pinched triangle, although both Surface Layout and Lit Surface stimulus factors affected reaction times. This finding runs counter to many previous studies where behavioral measures were affected by regularity violations that went unnoticed by participants (e.g., Müller et al., 2013). The absence of any effect of layout deviant on target detection could be explained by ceiling effects on this easy task. Notably, the hit rate was nearly 100% across all participants. Participants may also not have felt time pressure to respond quickly, as they were instructed simply to respond before the following stimulus presentation. No strong conclusions about the behavioral correlates of surface layout irregularities can be made, due to the ease of the task and participants' focus on accuracy. 

Considering both ERP correlates and behavior, there was no evidence for an interaction between changes to surface layout regularities and the detection of an oddball surface. Although these processes are likely indirectly related in the visual hierarchy, this paradigm did not reveal this. We stress that the Target Presence comparison was unplanned, and the limited number of target trials may have been insufficient for detecting task-driven effects. Furthermore, it is not obvious exactly what differences should be expected between targets and nontargets with relation to layout deviance, as these trials differed in contour differences, detection processes, and motor preparation and response. 

Stimulus-specific responses were independent of the deviance effect

In the current paradigm, we investigated changes in layout expectancies, where the effects of these changes should be independent of any particular layout or stimulus effects. To show this, we accounted for layout deviance across both a layout featuring a foreground triangle and an amodally-completed triangle obscured by a second surface. A second factor affecting illumination of these two surfaces was also included, which varied on every trial, and so had no sequential regularities of its own. In combination, these two features produced four stimulus configurations that differed significantly from one another in low-level perceptual properties. Despite being easily categorized upon “global” perceptual features like contour completion and surface segmentation, they differed in terms of the presence of edges near the fovea, the contrast polarity of edges, and the degree of foveal and parafoveal illumination (Figure 1). 

Somewhat surprisingly, each stimulus configuration showed a unique visual-evoked signature which differentiated one from the others. For example, a triangle appearing in front of a lit plane triggered a markedly attenuated P1 following stimulus onset, a reduction not seen in any other configuration (see Supplementary Movie S2). Analysis of stimulus factors specifically was unplanned, but showed P2 differences from Surface Layout, and later differences due to surface lighting. The effect of Surface Layout in particular is explained by previously-described negative differences in amodally-completed contours (Murray et al., 2004). 

These unique signatures serve to demonstrate, firstly, that the visual system fully registered the perceptual properties of each configuration. This occurred automatically and in spite of the ongoing task performed by observers. The on-line task only granted relevance to the triangle's shape, but did not explicitly draw attention to any other perceptual features such as the relative depth of the surfaces or light source. More importantly to the logic of this mismatch paradigm, the neural responses to each stimulus were orthogonal to responses to a deviance in the sequence of stimulus layout, as reflected in the absence of meaningful interactions between Layout Deviance and stimulus factors. These factors (and specific perceptual features of the stimuli) can therefore be excluded as alternative causes of the putative layout deviance effect. 

Deviance effect was not graded by prior evidence

We hypothesized that the amount of repetitions of a particular layout contributing to regularity should affect expectations about upcoming stimuli. In Bayesian terms, greater evidence should revise and strengthen perceptual models of the visual environment, and when subsequently violated these should give rise to greater prediction errors (Friston, 2005). In the current context, we quantified the strength of these prior beliefs using the number of repetitions of a surface layout within a microsequence. This simple single-trial model of mismatch encoding is similar to the “linearly modulated stimulus change model” employed in earlier roving standard mismatch studies (Baldeweg, Klugman, Gruzelier, & Hirsch, 2004; Ostwald et al., 2012). More recent and detailed approaches to this question include dynamic causal modelling (Garrido et al., 2009), Bayesian surprise (Mars et al., 2008; Itti & Baldi, 2009; Ostwald et al., 2012), and precision-weighted prediction errors (Stefanics et al., 2018). 

Although a surface layout deviant produced a clear mismatch negativity, this signal was not graded with prior evidence. Although the voltage of layout deviants appeared graded with respect to prior evidence, this effect was possibly cancelled by a subtraction of layout repeats to control for microsequence repetition effects. Notably, layout repeats also showed a gradation from prior evidence in the windows of interest. On face value, the absence of an effect of prior evidence suggests that while the visual system detects changes in surface layout (Näätänen et al., 2011), it is not sensitive to the strength of recent sensory evidence which could inform the latter. This result contrasts with recent findings by Stefanics and colleagues (2018), where the amplitude of vMMN differences correlated with modeled precision-weighted prediction errors on each trial. Rather than concluding that higher-level vision does not operate in a Bayesian fashion, we propose several explanations for this discrepant finding. 

For one, the number of repetitions of a layout was here limited to between two and five, with the five-repetition condition left unanalyzed. This number is notably fewer than in other roving standard designs; by way of contrast, microsequences in previous roving standard studies featured at least five presentations (Stefanics et al., 2018), 2-36 repeats (Baldeweg et al., 2004), 2–16 repeats (Ostwald et al., 2012) and 1–11 tones (Sumner et al., 2018). Accordingly, the number of layout repetitions in the current instance may not have been sufficient to drive a detectable gradation in deviance response. 

Perhaps paradoxically, it is also possible that observers' expectations that the stimulus layout would change might increase across a microsequence. The likelihood of a layout repeat versus a layout deviant in this paradigm was approximately equal, although after five layout repetitions a layout deviant always occurred. Having an awareness of an impending layout change could reduce the expectation of further layout repeats, despite the fact that conditional probabilities of deviance did not change across presentations. By counteracting the accumulation of evidence for a given standarḑ the effect of a “surprising” layout deviant would diminish, contrary to theory. Again, the limited length of microsequences in this study could have prevented graded expectancies from forming, while simultaneously allowing the tracking of second-order regularities in microsequence patterns. 

A more intriguing possibility is that two presentations of a stimulus layout is enough to generate a stable, high-fidelity hypothesis of surface segmentation. Further presentations may not provide more meaningful evidence for updating models of object relations in space, with outdated information quickly discounted. Put another way, previous evidence for interpretation of a visual scene may decay more rapidly than for other regularities (Harrison, Bestmann, Rosa, Penny, & Green, 2011). Ostwald and colleagues (2012) showed that recent evidence is weighted more than less recent evidence in a somatosensory roving standard paradigm, with the ideal window for Bayesian integration of evidence being 5–10. Such “one-shot learning” is also consistent with the theorized effect of high-precision prediction errors (Moran et al., 2013). 

If “just one look” is indeed enough evidence to form a stable model about surfaces in the environment, it may not be surprising to find different properties of evidence accumulation and prior belief formation, in comparison with previous predictive coding findings. In the natural environment, observers must quickly bind objects to locations to interact with them, and do not expect objects to change location in depth or switch positions in unlikely ways. Object relations like occlusion remain invariant over short time scales and minor changes in perspective, in contrast to lower-level visual properties like orientation, luminance, and retinal size. It is likely then that prior-updating for higher-level visual features, while also operating in a Bayesian fashion, differs in evidence weighting and temporal properties. 

Limitations

The results of this study should be considered in light of some limitations. For one, participants were advised to fixate at the center of the stimulus display during the procedure, but this was neither tracked nor controlled. Some participants could therefore have followed the surface “inducers” in hopes of making the detection task easier. Changes to local features of inducers (contour presence and absence, contrast polarity reversals) might then explain the prediction error signal, without the need for visual surface processing. Although we cannot exclude this possibility with the available evidence, there are several reasons why it is unlikely that participants would use such a strategy. First, the layouts and inducers appeared within parafoveal vision, with inducer centers reaching 3.4° eccentricity. Tracking an inducer that disappeared, moved, and reappeared each second would likely be more cognitively demanding than maintaining central fixation over an hour-long session. Furthermore, there is previous evidence that attending to inducers would not necessarily provide a task benefit: Ringach and Shapley (1996) had participants perform a similar fat/thin surface discrimination task, and performance suffered when observers attended to inducers rather than the whole array. Ceiling performance in the current results suggests that participants found the task very easy as designed. Considering these, it seems unlikely that participants would employ such a strategy in this task, against instruction. If participants did fixate away from center incidentally, EEG data with muscle activity generated by the associated eye movements would have been excluded from analysis at an early stage. 

A second theoretical limitation involves an assumption that observers processed the stimulus configurations as planar surfaces arranged in 3D space. An alternative interpretation is that the stimulus layouts differed in whether they included a real versus illusory (amodal) triangle; that is, participants may not have subjectively perceived Plane-Front configurations as including an occluded triangular surface. In this view, manipulations of stimulus layouts would drive the presence or absence of amodal illusory contours, but not changes in surface segmentation per se. As in the standard Kanizsa figure, this explanation does not require visual processing at the level of surfaces in three-dimensional space. Nevertheless, it would still involve amodal contour completion, a process itself shown to be feedback-driven and involving higher-level visual areas (Rubin, 2001; Kogo, Strecha, Van Gool, & Wagemans, 2010; Wyatte, Jilk, & O'Reilly, 2014) such as the lateral occipital cortex (Stanley & Rubin, 2003). Under both interpretations, the current findings demonstrate the tracking of feature regularities in a hierarchical visual process, as predicted by theories of predictive coding (Rao & Ballard, 1999). Because the present design confounded contour completion and surface occlusion, further research will be needed to establish whether perceptual predictions are occurring at the level of boundary completion or surface segmentation. 

Implications and future directions

The current research contributes to existing literature by showing the visual mismatch response can be elicited by surface segmentation deviance. To our knowledge, this research represents the first demonstration of mismatch effects driven by occlusion of objects (Winkler & Czigler, 2012). This adds to growing literature showing processing of regularities in high-level visual features. In particular, the tracking of irrelevant object-to-object relations has been evidenced by electrophysiological responses that were remarkably similar to the above findings (Müller et al., 2013). Other studies have also revealed that patterns of complex face dimensions, including emotional expression (Astikainen et al., 2013; Stefanics et al., 2018) and gender (Kecskés-Kovács et al., 2013), are registered by the visual system, even when face identities change. Considered in combination with this recent work, it appears that the visual hierarchy processes several types of high-level perceptual regularities, providing additional support for recent theories positing that predictive coding occurs at multiple hierarchical levels in perception (Garrido et al., 2008; Rao & Ballard, 1999). Future research in predictive coding in the visual system could investigate whether other regularities of object and scene perception are tracked, such as 3D structure, illumination constancy, and Gestalt grouping properties (Winkler & Czigler, 2012). 

In this study we failed to find an expected effect of recent perceptual evidence on irregularity detection, here codified as Prior Evidence. No gradated response from the number of immediately preceding repeats was observed, which contrasts with other work in perceptual predictive coding. As few repeats occurred in microsequences, strong conclusions cannot be drawn about this null effect. A future extension of the present study could further extend layout repetitions, into at least the tens of seconds. Furthermore, more advanced modeling (Garrido et al., 2009; Stefanics et al., 2018) may be more sensitive in this regard, and could also take into account the effect of incidental regularity changes across stimulus presentations, such as luminance and edge polarity reversals. Finally, we can expect that perceptual evidence “volatility” and processes of prior formation should vary for different classes of visual features, which could also explain the null effect of prior evidence here. Future research should employ trial-based computational models of prediction errors to explore predictive coding properties across multiple levels of the visual hierarchy (de Lange, Heilbron, & Kok, 2018), as recently done with color and emotion (Stefanics et al., 2018). 

Conclusion

We have shown through electrophysiology that the visual system is not only sensitive to the relative position of objects in space, but automatically tracks regularities in these layouts and is sensitive to their sudden change. Similar to growing research on other complex visual properties, layout deviance elicited a visual mismatch negativity peaking after 200 ms at posterior electrodes, and produced a central positive wave after 300 ms. In addition to being independent of task demands, these effects were invariant to lower-level stimulus properties and occurred for multiple variations of stimulus surface configuration. This mismatch response for surface perception provides further evidence that predictive processes occur across the visual hierarchy. 

Acknowledgments

This research was supported by a grant from the Faculty Research Development Fund at the University of Auckland to William G. Hayward. 

The authors thank Scott Claessens and Veema Lodhia for aid in analysis and data collection. The authors also thank Kara Federmeier, Meg Spriggs, Rachael Sumner, and Armin Saysani for helpful comments and feedback. 

This study was preregistered with the Open Science Foundation on November 3, 2017. This preregistration can be viewed at https://osf.io/qx4kr/. Some terminology used to describe the study has been changed; all other variations from planned experimentation and analysis have been noted already. 

The data that support the findings of this study are available from the corresponding author, MO, upon reasonable request. 

Commercial relationships: none. 

Corresponding author: Matt Oxner. 

Email: . 

Address: School of Psychology, University of Auckland, Auckland, New Zealand. 

Supplementary Materials

Supplementary Movie S1. Demonstrates the experimental stimuli as presented. 

Supplementary Movie S2. Illustrates analysis of ERP responses to stimuli in time and space.