Early-vision brain responses which predict human visual segmentation and learning

Abstract

Brain processes underlying visual segmentation have been widely studied, being part of the basic processes underlying perception. However, the underlying constraints on perceptual thresholds, set by neuronal processing, remain unclear. Here, the relationship between human visual performance and brain activity was examined using the backward-masked texture segmentation task. Performance showed dependence on the time-interval between target and mask as well as on the amount of prior practice. Correspondingly, early components of human event-related potentials (ERPs) recorded over occipital electrodes showed strong interactions between target and mask responses, suggesting interference with perception processes when the presented mask interacts with sustained target processing. These interactions, revealing an otherwise undetected extended processing time course beyond the early component of the target response, enabled us to quantify individual neuronal thresholds that closely matched the behavioral thresholds ( r = 0.93, p = 0.00003). Furthermore, these neuronal thresholds could be improved by practice, suggesting neuronal mechanisms affected by perceptual learning. Predicting performance level not directly detected in the ERP but rather by further interactions shown here in early stages of the visual hierarchy may have important implications in the study of human perception. Practice seems to reduce the temporal interactions between the successive stimuli, revealing brain processes underlying perceptual learning.

Introduction

Separating objects from background is one of the basic processes underlying our conscious perception in the visual domain, referred to as image segmentation (Julesz, 1965). While studies of human performance lead to an account which is mostly based on low-level “effortless” sensory processing (Julesz, 1981; Rubenstein & Sagi, 1990), more recent studies show that processes of perceptual learning may result in improvements of perceptual thresholds (Karni & Sagi, 1991). These studies have shown that performance drops when the time-interval between the target and the following mask stimulus (stimulus onset asynchrony, SOA) is decreased (Karni & Sagi, 1991, 1993; Karni, Tanne, Rubenstein, Askenasy, & Sagi, 1994; Mednick, Nakayama, & Stickgold, 2003; Schwartz, Maquet, & Frith, 2002; Stickgold, James, & Hobson, 2000a; Stickgold, Whidbee, Schirmer, Patel, & Hobson, 2000b). Furthermore, these studies have shown that latent, slowly evolving, between-session improvement often occurs. It was suggested that neuronal changes in early stages of the visual system underlie these performance gains which were specific to basic features of the stimulus (Karni & Sagi, 1991), with sleep having an important role in mediating these gains (Censor, Karni, & Sagi, 2006; Karni et al., 1994; Schwartz et al., 2002; Stickgold et al., 2000a, 2000b). Theories of vision supported by single-cell recordings in non-human primates (Lamme, Zipser, & Spekreijse, 2002), suggest the involvement of slower, possibly top-down, processes in this early visual function. 

Previous studies were not consistent regarding the electrophysiological correlates of human segmentation, suggesting a range of cortical areas including early visual areas such as primary visual cortex (V1) and higher fronto-parieto-temporal activations (Boehler, Schoenfeld, Heinze, & Hopf, 2008; Del Cul, Baillet, & Dehaene, 2007; Fahrenfort, Scholte, & Lamme, 2007). However, these studies, while using the backward masking method, failed to isolate the temporal elements of the stimuli and used indirect complex analysis of differential ERP or ERMF (event-related magnetic field) responses, usually showing small-scale effects of group averages (Bach & Meigen, 1998; Boehler et al., 2008; Del Cul et al., 2007; Fahrenfort et al., 2007). These studies were therefore unable to directly quantify the individual perceptual thresholds related to the task, based on the measured brain responses. 

Therefore the main aim of the current experiments was to study whether and to what extent does early-vision processing revealed by electrophysiological human brain responses predict behavioral texture discrimination performance and learning. We measured human perceptual thresholds while recording ERPs, before and after learning of a visual discrimination task closely manipulated over a wide temporal scale. We were able to find “neuronal response thresholds” measured during early-vision processing, and reveal an extended processing time-course, otherwise undetected, beyond the early observed components of the response. These brain responses recorded over occipital electrode sites could individually predict perceptual thresholds of human observers, before and after perceptual learning. 

Materials and methods

Subjects

The subjects were 7 paid undergraduate students with normal or corrected-to-normal vision. An informed consent form was obtained from all subjects, in accordance with the approved Declaration of Helsinki for this project. 

4 subjects participated in the main learning experiment. Due to possible effects of prior experience, these subjects were naïve, with no prior experience in the task. Each subject performed 2 identical sessions separated by sleep, except one subject that performed an additional 3 ^rd session. Low quality EEG signal in the second session of one of the subjects was excluded from analysis. 

3 additional subjects performed a single session and were added to the correlation analysis between discrimination and ERP thresholds (see Figure 4). 

Apparatus

The stimuli were presented on a 19″ Mitsubishi Diamond Pro 930SB color monitor, using a PC with an Intel Pentium processor. Monitor refresh rate was 100 Hz. The luminance of the stimulus (line textures) was 64 cd/m ² in an otherwise dark environment. Subjects used a chin-rest to maintain a stable position. 

Stimuli and task

The standard texture stimuli was used (Censor et al., 2006; Karni & Sagi, 1991; Stickgold et al., 2000a), consisting of a target frame, which appeared for 40 ms. The target was followed by a patterned mask which appeared for 100 ms, as shown in Figure 1. Observers had to decide whether an array of 3 diagonal bars embedded in a background of horizontal bars (19 × 19, 0.5° × 0.035° each, and spaced 0.72° apart) was horizontal or vertical. Display size was 14° by 13.5° of visual angle, viewed from a distance of 100 cm. The target appeared randomly and equally, either in the lower left or lower right visual quadrant, with its center at 5.3° of visual angle from center of display. Fixation was enforced by a forced-choice letter-discrimination task, between a “T” and an “L”, at the center of the display. The time-interval between the target stimulus onset and the mask (stimulus onset asynchrony, SOA) was manipulated. Each session started at an SOA of 340 ms, which was gradually decreased by SOA-dependent steps of 20–40 ms of 3 blocks of trials per SOA. In the main learning experiment, each block contained 26 trials (78 trials per SOA, ∼900 trials per session). 3 additional subjects performed a single session with blocks of 50 trials (150 trials per SOA, ∼1350 trials per session). Each psychometric curve obtained was fitted with the Weibull function, with an additional finger error parameter 1 − , yielding the function: 

$P\left(t\right)=p\{1-\frac{1}{2}\exp [-{\left(\frac{t}{T}\right)}^{\beta }]\}+\frac{1-p}{2}=\frac{1}{2}\{1+p[1-\exp [-{\left(\frac{t}{T}\right)}^{\beta }]]\}$

(1)

where T is the threshold for each curve, defined as the SOA for which 81.6% of responses were correct when p = 1. In each session, the threshold SOA for the left and right targets was averaged. Sessions were terminated when the subject reached an SOA with close to chance level of performance (defined as less than 65% correct responses, the lowest SOA reached was 50 ms). Subjects additionally performed blocks of trials containing only the target stimulus, serving as baseline for the ERP analysis as detailed below. 

Figure 1

 

Trial sequence of the texture discrimination task. Subjects fixate at a circle in the center of the screen and activate the sequence: (a) a 40 ms target consisted of 3 diagonal bars at the lower left or lower right quadrant of display, differing only in orientation from a background of horizontal identical bars. Subjects have to discriminate whether the 3 bars were vertical or horizontal, while a small rotated letter (T or L) at center of display serves as a fixation task; after a blank inter-stimulus interval appears (b) a 100 ms mask made of randomly oriented V-shaped micro patterns and at the center, a pattern of superimposed T and L; finally, there is a blank screen until response. The time-interval between the target onset and the mask (SOA) was gradually decreased within the session.

EEG recording and analysis

The EEG signals were recorded continuously by using a Biosemi Active II system with 64 Ag-AgCl pin-type active electrodes mounted on an elastic cap according to the extended 10–20 system, and from two additional reference electrodes placed at the right and left mastoids. Recordings were sampled at 512 Hz. Digital 0.1–250 Hz bandpass filter and 50 Hz notch filter (filtering 49–51 Hz) were used. Epochs with EEG signal exceeding ±100 μV were removed from the main analysis of the Oz electrode signal. Recordings were time-locked to target visual stimulus onset. Individual VEPs were then averaged across trials for each SOA separately, using a baseline correction over 1000 ms pre stimulus. 

In order to quantify the averaged amplitude of the N1 component in the mask response at each SOA, the averaged target-alone response, serving as baseline, was subtracted from the average target-and-mask response at each SOA (averaging was over 78–150 trials). Then, the magnitude (absolute amplitude) of the averaged N1 mask was calculated by averaging the amplitude of the ERP in a time-window of SOA + 100 ± 50 ms:

$\frac{\sum _{t=SOA+50}^{t=SOA+150}{V}^{\prime }\left(t\right)}{100}$

, where V′( t) is the average waveform after subtracting the averaged target-alone ERP. This time period does not include a small overshoot prior to the N1 response, which appeared in one of the subjects. 

Finally, to obtain the mask's neurometric curve for the full session (normalized amplitude between 0 and 1 as a function of all the SOAs in this session), for each SOA the calculated averaged mask response amplitude was normalized with the largest averaged mask response amplitude in this session. To obtain the target's N1 neurometric curve for the full session, for each SOA the peaks of the target response were normalized with the largest target's peak. The longest SOAs were not included in the curve if the differential mask responses after normalization were smaller than 0.75, possibly due to a strong interaction with the P300 target peak. 

In order to determine the neuronal threshold for each session, each neurometric curve obtained was fitted with the Weibull function described above:  

$M\left(t\right)=\frac{1}{2}\{1+p[1-\exp [-{\left(\frac{t}{T_{EEG}}\right)}^{\beta }]]\},$

(2)

where T _EEG is the threshold for each curve and M( t) is the computed differential normalized mask signal, at each SOA t. 

Results

Psychophysics

The experiment was designed with session structure previously shown to yield between-session learning (Censor et al., 2006). Four subjects performed two morning sessions in two separate days, with blocks containing 26 trials each (78 trials per SOA). Three additional subjects performed a single session with blocks containing 50 trials each (150 trials per SOA). 

All subjects ( n = 4) improved between training sessions, showing a significant between-session threshold improvement (mean threshold improvement of 16.9 ms, SE ± 4.4 ms, paired t-test: p = 0.015), as shown in Figure 2. Subject ZT who performed an additional third training session, continued to improve 7.5 ms between the second and third session. 

Figure 2

 

Between-session improvement of discrimination thresholds. (a) Average thresholds of subjects ( n = 4) on each of the practice days, showing significant between-session learning. Error bars represent standard errors. (b) Individual thresholds of the subjects, on each of the practice days.

These results are consistent with previous studies showing between-session learning in the texture discrimination task, when sleep is afforded between sessions (Karni et al., 1994; Stickgold et al., 2000a, 2000b), and are comparable with between-session gains obtained using the same number of trials (mean overnight gains of 22.4 ms, SE ± 3.6 ms, see Censor et al., 2006). 

ERPs

The main goal of the experiments was to study whether the event-related potentials correlate with the level of texture discrimination performance and learning. Figure 3a displays the scalp distribution of the amplitude of the N1 response (averaged in a time-window of 100 ± 50 ms from stimulus onset) to the target alone (no mask presented, no epochs were excluded). This strong and fast negative response may result from the high intensities used, as strong intensity increments are known to produce strong perceptual masking and strong inhibitory effects at the primary visual cortex (Tucker & Fitzpatrick, 2006). Furthermore, responses are known to depend on the stimulated visual field and stimulus type, locations and physical parameters such as luminance, contrast and spatial frequency (see for example Vassilev, Mihaylova, & Bonnet, 2002). 

Figure 3

 

Texture stimuli ERPs (a single subject example). (a) Scalp distribution of the averaged N1 response in a time-window of 50–150 ms from target onset when the target was presented alone; the color bar indicates the average amplitude in microvolts. (b) The response at the strongest Oz electrode (indicated by the arrow) plotted for the different SOAs. The decrease in the mask response with decreasing SOAs is clearly visible. (c) The psychometric (blue) and neurometric (red) curves of the same subject, showing with decreasing SOAs the same pattern of decreased correct discrimination and differential normalized N1 mask response. In contrast, the N1 target peak response (dashed) remained nearly constant with decreasing SOAs.

Figure 3b shows the ERP time-locked to target onsets at the occipital electrode Oz (indicated by an arrow) during the different SOA conditions. The graph clearly shows that while the target's response remained constant, the task-irrelevant mask response decreased in amplitude with decreasing SOAs. 

In order to quantify the averaged N1 mask response amplitude for each SOA, the average target-alone baseline waveform was subtracted from each target-and-mask waveform (at each SOA). Then, for each SOA the average N1 mask response was calculated. Finally, for each session of decreasing SOAs, the calculated mask responses for each SOA were normalized to the SOA with the largest mask response amplitude to obtain the neurometric curve. Figure 3c displays the psychometric curve of a single subject, along with the neurometric curve of differential N1 mask responses, showing the same pattern of decrease in correct discrimination and response to the mask, with decreasing SOAs: as the mask response dropped, discrimination deteriorated. The results further show that the target N1 peak response remained nearly constant across decreasing SOAs. Further analysis methods such as computing differential target responses at different SOAs using the target-alone response as baseline, did not show any consistent pattern of decreasing target responses with decreasing SOAs across all subjects. These results suggest that discrimination is affected by the same temporal regime of the ERP response in which the mask is affected, implying interference with further sensory processing of target, up to 150 ms beyond the robust target N1 transient evoked response. This interference with target processing may involve temporal integration of target and mask, thus reducing the response to the mask. These results further show that, for our stimuli, the early component of the target ERP waveform is not sensitive to the full temporal course of target processing, though as shown above, it seems that the underlying hidden process can be probed with the delayed mask. 

In order to determine whether the ERP correlated with behavior, for each subject in each session the ERP thresholds were calculated from the neurometric curves of the differential N1 mask responses. Surprisingly, these thresholds were very similar to the psychophysical thresholds, with the ERP thresholds biased downward by 0–30 ms as can be seen in Figure 4 (the exact relationship between the thresholds may depend on the estimation method used). The results showed a high linear correlation between the psychophysical correct discrimination thresholds and the ERP differential N1 mask response thresholds ( r = 0.93, p = 0.00003, see Figure 4). Our steep threshold functions predict large changes in percent of correct responses as a result of a small change in ERP response around threshold, accordingly we found no significant differences between the ERP differential N1 mask responses related to correct and incorrect trials, around threshold. This is in line with previous studies showing small-scale differences between large averaged populations of single trials (Boehler et al., 2008; Del Cul et al., 2007; Fahrenfort et al., 2007). Therefore, the predictive power of our results lies in our ability to extract the “neuronal thresholds,” which show that behavior can be largely predicted by the ERP signal, using the N1 response to the mask stimulus. Integration of the mask processing with the target processing may result in decrease of correct discrimination, explaining the correlation between discrimination thresholds and ERP N1 mask thresholds affected by target-mask interactions. 

Figure 4

 

Correlation between discrimination thresholds and ERP thresholds. Psychophysical correct discrimination thresholds as a function of ERP N1 mask responses thresholds, showing a high linear correlation of r = 0.93, p = 0.00003. The legend shows subject initials and session number (except for the 3 subjects who performed a single session).

Between-session learning resulted in psychophysical threshold decrease ( Figure 2). Along with this learning, the ERP mask response thresholds decreased also between the two training sessions, as shown in Figure 4 (subject SG 9.3 ms, subject ZJ 24.9 ms ERP threshold decrease. Subject ZT showed a slight increase of 4.3 ms between the two sessions, but on the third session his ERP threshold decreased by 20.1 ms relative to the initial session). The decrease in ERP threshold resulting from increased mask responses suggests that learning reduces the temporal interactions between target and mask. 

Discussion

The results of this study show that performance level in the texture discrimination task is highly correlated with the early N1 mask ERPs recorded at occipital scalp sites positioned over early visual areas. This correlation, possibly due to target-mask interaction, exposed a neural determinant of performance which operates beyond the robust early target response, though still relatively early compared to later components such as the P300 response. Our ability to extract these neuronal thresholds using target-mask interactions enables us to predict the behavioral thresholds. Furthermore, our results suggest that perceptual learning could improve these neuronal thresholds by reducing the temporal interactions between target and mask. 

Our uncommon strong and fast negative responses (instead of the common P100) may be a result of the high intensities we used. Strong intensity increments are known to produce strong perceptual masking and strong inhibitory effects at the primary visual cortex, shown in intracellular recordings from layers 2/3 (Tucker & Fitzpatrick, 2006). Overall, these potentials depend on the stimulated visual field and stimulus type, showing high variability between different studies using a variable range of stimuli-types, locations and physical parameters such as luminance, contrast and spatial frequency (see for example Vassilev et al., 2002). In particular, stimulus intensity has a major factor in setting the delay of the specific events. Therefore, our stimuli which present a strong intensity increment to the visual system, may explain the fast negative responses observed here (when we used the same stimuli with lower intensity, we indeed obtained a more common delayed negative peak at around 130 ms, of smaller magnitude; results not shown). 

Previous studies have shown similarity of early brain responses across several visual dimensions of texture segregation such as luminance, orientation, motion and stereo (Bach & Meigen, 1997). A negative 110 ms occipital peak response was shown, specific to texture segregation and unaffected by attention manipulations (Heinrich, Andrés, & Bach, 2007). Other studies have found pronounced differences between the segregation-specific potentials evoked by different visual features (Fahle, Quenzer, Braun, & Spang, 2003). Evidence from ERP studies in both monkeys and humans suggests that texture segregation involves processing by the primary visual cortex (Lamme, Van Dijk, & Spekreijse, 1992). Our results show that the N1 mask response decreases with decreasing SOAs (Figure 3b), while interestingly, the target N1 transient response is unaffected. The correlation with performance suggests that performance fails when the presented mask is integrated with the target, limiting its effective processing, thus revealing an extended processing time-course of up to 150 ms beyond the transient target response, though still relatively early compared to later components such as the P300 response. Integration of the mask with the preceding target, may explain the decreasing response to the mask signal: The reduced N1 mask response may be the response to elements in the mask stimulus that are not overlapping with the target and therefore not integrated with it, and processed in alternative networks. These results therefore support masking by integration theories, postulating that the mask influences the representation of the preceding target due to integration of the mask with the preceding stimulus, resulting in a representation in which the features of the mask merge with the features of the target (see for example Coltheart & Arthur, 1972). 

The decrease in mask response is consistent with previous results (Musselwhite & Jeffreys, 1983) which used a simple double-pulse pattern presentation to show that the second pattern of the stimulus pair evokes a consistently smaller C1 component measured at occipital electrodes, than the first pattern does. Furthermore, Musselwhite and Jeffreys (1983) showed that comparable limiting SOA values are needed for the C1 components of the two patterns to separate out forming a double-peaked response, and for the stimulus to be perceived as two distinct events. Another study (Harris & Nakayama, 2007) found face-selective decrease in evoked MEG response to the second of two temporally adjacent stimuli, relating it to adaptation processes. Decrease in amplitude of response to the second of two rapidly successive stimuli was also shown in the auditory domain, and correlated with performance decrease in an auditory discrimination task performed by poor readers (Nagarajan et al., 1999). 

Recordings from primary visual cortex of awake monkeys (Lamme et al., 2002) showed that initial response transients, signaling basic features of the target such as orientation were not affected by decreasing target-mask intervals which reduce performance. However, V1 figure-ground signals were selectively suppressed at target-mask intervals that resulted in reduced performance. Our current study, implying processing beyond the transient target response therefore extends these results to humans, using the texture discrimination task which requires figure-ground segregation. Other ERP studies in humans showed correlation between behavioral accuracy in a natural scene categorization task and differential ERP activity between target and distractor responses (Bacon-Macé, Macé, Fabre-Thorpe, & Thorpe, 2005), both affected by target-to-mask delay. Previous studies correlating electrophysiological measurements from early visual areas with texture segregation have usually analyzed group averages and used indirect differential ERP or ERMF response analysis, such as subtraction of figure from non-figure trials or differences between correct and incorrect trials, usually giving small-scale effects (Bach & Meigen, 1998; Boehler et al., 2008; Fahrenfort et al., 2007). 

The results showing high correlation of the ERP N1 mask signal with behavior imply that performance in a visual task such as texture discrimination depends in early visual areas on the temporal interactions between early target and mask responses. Moreover, the results presented here show that practice in a visual task such as texture discrimination, seems to reduce the temporal interactions between target and mask. Texture segmentation studies further support the notion of learning as separating the task-relevant signal stemming from the texture from the task-irrelevant signal of the mask (Schubö, Schlaghecken, & Meinecke, 2001). Previous studies have shown reduction of early electrical activity of trained stimuli (Pourtois, Rauss, Vuilleumier, & Schwartz, 2008), in agreement with the view that learning biases intracortical inputs in favor of inhibitory activity to decrease interference from background elements (Karni & Sagi, 1991; Pourtois et al., 2008; Tsodyks & Gilbert, 2004). Such inhibition of background elements reduces the sustained activity of the processing network, possibly reducing its duration, enabling better separation between the response to the target and the response to the following mask. This may explain why following learning, the N1 mask response decreases less, resulting in improvement of the neurometric ERP mask thresholds along with the improvement of the behavioral discrimination thresholds. Enhanced inhibition may also be consistent with increments of BOLD signal in V1 as a result of perceptual learning as shown in previous studies (Schwartz et al., 2002; Yotsumoto, Watanabe, & Sasaki, 2008). 

Previous studies using the texture discrimination task (Censor et al., 2006; Censor & Sagi, 2008; Karni & Sagi, 1991, 1993; Karni et al., 1994; Mednick et al., 2003; Schwartz et al., 2002; Stickgold et al., 2000a, 2000b), showed that latent, slowly evolving, between-session improvement often occurs. These studies suggested that neuronal changes in early stages of the visual system underlie these performance gains. While most of these behavioral studies based this conclusion on specificity of learning to basic properties of the stimulus (but see Mollon & Danilova, 1996, for counter arguments), our results showing correlation with performance of early ERP components recorded at scalp locations over early visual areas provide direct electrophysiological evidence further supporting the role of early visual areas in performance and learning of texture discrimination. Previous studies have also shown that within session (Casco, Campana, Grieco, & Fuggetta, 2004) and between-session (Pourtois et al., 2008) learning of texture segmentation modulates electrophysiological responses. While most such studies showed small-scale modulations as a result of learning, the pronounced effects found here which highly match the behavioral outcome of individual subjects do not rule out smaller electrophysiological modulations, but do enable to predict the behavioral thresholds and suggests that perceptual learning could improve neuronal thresholds by reducing the temporal interactions between target and mask. This account does not contradict with previous studies showing retinotopically-local texture learning (Karni & Sagi, 1991), since additional target-dependent neuronal processes such as processes involving top-down spatial attention (Martínez et al., 1999) could additionally modulate cortical activity and enhance different learning mechanisms. Involvement of additional learning mechanisms is further supported by our latest study using the texture discrimination task, showing that training-induced resistance to perceptual adaptation generalizes to untrained retinal locations (Censor & Sagi, 2009). 

Previous studies suggested that practice sessions with a texture stimulus may have a role in consolidating effective network connectivity, serving as a memory trace (Censor & Sagi, 2008). The modified network with increased efficiency may improve temporal separation by renormalizing connectivity patterns (Huber, Ghilardi, Massimini, & Tononi, 2004; Tononi & Cirelli, 2006) or reducing local energy demands by extending network size (Karni et al., 1995). Models of cortical reorganization involving temporal sharpening could also account for reduction of target-mask temporal interactions as part of a learning process (Moldakarimov, McClelland, & Ermentrout, 2006). Previous studies have shown that trained monkeys exhibited increase in temporal coherence or sharpness of the neural response, which correlated with improvement in a frequency discrimination task (Recanzone, Merzenich, & Schreiner, 1992). 

Our results show that behavioral performance can be predicted from early ERP signals obtained from early visual areas. Performance seems to fail when the presented mask interferes, possibly by integration, with the effective processing of the target signal, revealing an extended processing time-course beyond the transient target response. The correlation found here may have an essential role in the underlying mechanism limiting visual perception, such as backward masking. Perceptual learning in the texture discrimination task may therefore involve reduction of the temporal interactions between the target and the mask. Predicting performance level not detected in the transient target response but rather by further interactions shown here, may have important implications to the study of human perception, and could be used to study further limitations on perception including other sensory tasks. 

Acknowledgments

This work was supported by the Basic Research Foundation administered by the Israel Academy of Sciences and Humanities. 

Commercial relationships: none. 

Corresponding author: Dov Sagi. 

Email: [e-mail]. 

Address: Weizmann Institute of Science, Rehovot 76100, Israel.