Visual perceptual learning of a primitive feature in human V1/V2 as a result of unconscious processing, revealed by decoded functional MRI neurofeedback (DecNef)

Abstract

Although numerous studies have shown that visual perceptual learning (VPL) occurs as a result of exposure to a visual feature in a task-irrelevant manner, the underlying neural mechanism is poorly understood. In a previous psychophysical study (Watanabe et al., 2002), subjects were repeatedly exposed to a task-irrelevant Sekuler motion display that induced the perception of not only the local motions, but also a global motionmoving in the direction of the spatiotemporal average of the local motion vectors. As a result of this exposure, subjects enhanced their sensitivity only to the local moving directions, suggesting that early visual areas (V1/V2) that process local motions are involved in task-irrelevant VPL. However, this hypothesis has never been tested directly using neuronal recordings. Here, we employed a decoded neurofeedback technique (DecNef) using functional magnetic resonance imaging in human subjects to examine the involvement of early visual areas (V1/V2) in task-irrelevant VPL of local motion within a Sekuler motion display. During the DecNef training, subjects were trained to induce the activity patterns in V1/V2 that were similar to those evoked by the actual presentation of the Sekuler motion display. The DecNef training was conducted with neither the actual presentation of the display nor the subjects’ awareness of the purpose of the experiment. After the experiment, subjects reported that they neither perceived nor imagined the trained motion during the DecNef training. As a result of DecNef training, subjects increased their sensitivity to the local motion directions, but not specifically to the global motion direction. Neuronal changes related to DecNef training were confined to V1/V2. These results suggest that V1/V2 are involved in exposure-based task-irrelevant VPL of local motion.

Introduction

Visual perceptual learning (VPL) refers to a long-term performance enhancement resulting from visual experiences (Sagi, 2011; Watanabe & Sasaki, 2015) and can occur by mere exposure to a visual feature, a phenomenon referred to as exposure-based VPL (Arsenault & Vanduffel, 2019; Frank et al., 2021; Galliussi, Grzeczkowski, Gerbino, Herzog, & Bernardis, 2018; Gutnisky, Hansen, Iliescu, & Dragoi, 2009; Lorenzino & Caudek, 2015; Murris, Arsenault, Raman, Vogels, & Vanduffel, 2021; Pascucci, Mastropasqua, & Turatto, 2015; Protopapas et al., 2017; Rosenthal & Humphreys, 2010; Seitz & Watanabe, 2003; Watanabe et al., 2002; Watanabe, Nanez, & Sasaki, 2001). 

Multiple psychophysical experiments have been conducted to infer the mechanism of exposure-based VPL (Frank et al., 2021; Galliussi et al., 2018; Tsushima, Seitz, & Watanabe, 2008; Watanabe et al., 2002; Watanabe et al., 2001). In one of these studies, Watanabe et al. (2002) exposed subjects to a peripherally presented visual motion display, termed a “Sekuler display”, consisting of dots moving randomly within a certain range of directions as a task-irrelevant stimulus. This display induced the perception of not only the local dots’ motions, but also the global motion corresponding with the spatiotemporal average of the local motion vectors (Williams & Sekuler, 1984). During the exposure, subjects conducted a different task in the center of the display. After repeated exposure to the Sekuler display, subjects’ sensitivity within the direction range of the exposed local motion increased. In a different condition of this experiment, subjects were trained on a global motion discrimination task in the Sekuler display. The results of this condition showed that subjects’ sensitivity within the local dots’ direction ranges increased in the earlier phases of training. However, in later phases of training, sensitivity enhancements were observed in the global motion direction only if a task was conducted on the global motion. These findings raised the possibility that, in early phases of training, VPL, whether task-relevant or task-irrelevant, occurs for local motions in early visual areas, whereas in later phases, VPL develops based on task-relevant global motion in higher-level stages of visual processing. 

A previous study using functional magnetic resonance imaging (fMRI) in human subjects found that V1 is the earliest visual cortical area to respond to local motions in the Sekuler display whereas, V3A is the earliest visual cortical area to respond to global motions (Koyama et al., 2005). However, this finding alone does not indicate that V1 is involved in VPL of local motions. First, although V1 is the earliest area of visual cortex to respond to local motions, it is highly likely that not only V1, but also higher-level areas within and outside visual cortex respond to local motions. Second, it is unlikely that all of the areas that respond to local motions are involved in VPL of local motions. 

To examine more directly whether early visual areas are involved in exposure-based VPL of local motions, we conducted an experiment using fMRI decoded neurofeedback (DecNef) training in human subjects. A number of studies have found that the repetitive induction of fMRI activation patterns similar to those evoked by a real stimulus without presentation of the stimulus by means of DecNef training, can enhance the performance on the stimulus (Amano, Shibata, Kawato, Sasaki, & Watanabe, 2016; Shibata, Watanabe, Sasaki, & Kawato, 2011; Watanabe, Sasaki, Shibata, & Kawato, 2017). 

Using DecNef training, we found that the repetitive inductions of activation patterns in V1/V2, similar to those evoked by the Sekuler display, resulted in performance enhancement of the local motion directions and not specifically of the global motion direction to which V3A is the earliest area to respond (Koyama et al., 2005). The fMRI results showed that only the V1/V2 activation patterns were largely correlated with the neurofeedback training. Subjects’ postexperimental reports indicated no evidence that they had perceived or imagined any motion during the DecNef training. Taken together, these results are in accordance with the hypothesis that exposure-based VPL of primitive local motion features involves early visual areas. 

Methods

Subjects

A total of 18 subjects (eight females) with normal or corrected-to-normal vision participated in the study. Subjects had never participated in previous visual training experiments and were naïve or had little knowledge about brain anatomy. Eight subjects participated in the neurofeedback experiment (mean age, 23.4 ± 2.4 years). Ten subjects participated in the control experiment. However, data from two subjects in the control experiment were excluded from further analyses because their ages were outside the age range of the subjects in the neurofeedback experiment. Therefore, analyses were conducted on data from the remaining eight subjects in the control experiment (mean age, 26.3 ± 13.2 years). There were no differences in recruiting standards between the neurofeedback and control experiments. All subjects gave informed written consent to participate in the study, which was approved by the Institutional Review Board of Brown University adhering to the Declaration of Helsinki. 

Neurofeedback experiment

Experiment timeline

The neurofeedback experiment consisted of four stages, as shown in Figure 1A: the pretest stage (1 session), fMRI motion decoder construction stage (1 session), fMRI neurofeedback training stage (3 sessions), and posttest stage (1 session). The pretest and posttest stages were conducted outside the scanner. Sessions were spaced by an interval of at least 24 hours. 

Figure 1.

 

Experimental design and stimuli. (A) Timeline of the neurofeedback experiment. The decoder construction and neurofeedback training stages were conducted inside the MRI scanner (gray background). The pretest and posttest stages were conducted psychophysically outside the scanner. Each session was conducted on a separate day. (B) Sekuler motion display. Local motions were within ±30° range of the global motion direction. The small arrows attached to each dot show the motion direction of this dot for illustrative purposes only and were not shown in the real experiment. The large red arrow shows the global motion direction. (C) Example trial during pretest and posttest. Subjects responded by pressing the number on the keypad corresponding with the coherent motion direction and were given no time limit for the response. (D) Illustration of the nine motion directions used for pretest and posttest. The motion directions included the global motion direction, the local motion ranges (from −30° to +30° relative to the global motion direction) and other motion directions outside the range of the Sekuler motion display (from −60° to −30° and from +30° to +60° relative to the global motion direction). (E) Timeline of the control experiment. All sessions including the decoder exposure session were conducted psychophysically outside the scanner. There was a 3-day-long break between the decoder exposure session and the posttest to mimic the time-interval required by the neurofeedback training sessions in the neurofeedback experiment (see A). Each session was conducted on a separate day.

Statistical analysis

The normality of the data was checked with the Shapiro-Wilk test. All datasets were normally distributed. Therefore, analysis of variance (ANOVA) and two-tailed t tests were applied to determine the statistical significance of the behavior measures. Effect sizes were also calculated accompanying the statistical significance (Cohen's d for the t test). All statistical analyses were performed using MATLAB (MathWorks, Natick, MA) and SPSS 22 (IBM Corp., Armonk, NY). Shapiro–Wilk tests were performed with MATLAB functions developed by Öner and Deveci Kocakoç (2017). 

Motion stimuli

Two types of visual motion displays were used. One was a coherent motion display (Braddick et al., 2001; Salzman, Murasugi, Britten, & Newsome, 1992). The other was a Sekuler display, with additional noise overlaid (Williams & Sekuler, 1984). 

During the pretest and posttest stages, a 10% coherent motion display was presented. The display consisted of 70 white dots moving within an aperture with a radius of 4.5°. For each frame, 10% of the dots (coherent dots) moved coherently in a predetermined direction. The coherent dots moved at a speed of 7.1°/s. The other dots were randomly placed at different locations from frame to frame. Each frame was presented for approximately 16.7 ms. See the Pretest and posttest section in this article for further details. 

During the fMRI motion decoder construction stage, a Sekuler display was presented as shown in Figure 1B. The motion display consisted of local motions moving spatiotemporally and randomly within a certain range (between −30° and +30° relative to the global motion direction). This type of display induces a global motion perception in the direction of the spatiotemporal average of the local motion signals (Koyama et al., 2005; Williams & Sekuler, 1984). The Sekuler display in the decoder construction session was perceptually different from the coherent motion stimuli in the pretest and posttest stages. For the Sekuler display stimuli, 35 signal dots and 65 noise dots moved within an aperture (4.5° radius) presented at the center of the display. Previous studies have shown that, with this percentage of signal dots, subjects are able to discriminate global motion directions with around 70% accuracy (Watanabe et al., 2002). For each frame, each signal dot moved in a random direction uniformly distributed within the range from a predetermined global motion direction, defined as the mean of the distribution of the local motion signals. Each noise dot moved randomly without any direction restriction. Subjects perceived both the local motion of each moving dot, as well as the global motion. Each frame was presented for 50 ms. Two ranges of Sekuler displays were used for each subject. The global motion direction in one range was rotated 120° from that in the other range so that there was no overlap between the two ranges of Sekuler displays. We did not rotate the Sekuler displays for 180° to avoid the 180° error (Bae & Luck, 2019), which means that subjects might confidently perceive the visual motion direction as 180° opposite to the true visual motion direction. One range of global motion direction corresponded with the trained motion range for neurofeedback. The other range was used for control (untrained). The trained global motion direction was pseudorandomly selected for each subject and counterbalanced across subjects in the neurofeedback and control groups. In the Results section, we realigned the motion directions across subjects as the trained motion direction. See the fMRI motion decoder construction section for more details. 

For each motion display, the background was black except for a white bull's eye on a gray disc with a radius of 0.75° presented at the center of the display. 

Apparatus

The visual stimuli for testing sessions outside the scanner were presented on an LCD monitor (1024 × 768 resolution, 60 Hz refresh rate), and visual stimuli during the MRI sessions were presented on an MRI-safe LCD display (BOLDscreen 32 LCD for fMRI, Cambridge Research Systems [Rochester, UK], 1920 × 1080 resolution, 120 Hz refresh rate). The visual stimuli were controlled via a Mac OS computer and Psychtoolbox (Brainard, 1997) outside the scanner and a Windows 7 computer and Psychtoolbox inside the scanner. 

Pretest and posttest

The purpose of the pretest and posttest stages was to measure the subjects’ sensitivity to each motion direction within or beyond the local motion direction range in the Sekuler display. To this purpose, subjects performed a coherent motion direction discrimination task, which is explained in detail below in this article. The sessions were conducted in a dimly lighted room. A total of 18 motion directions was tested in each session. For each direction a total of 20 trials was performed. In each session, there were two blocks; each block presented either the trained or the untrained global motion direction as well as 8 other motion directions (thus, there were a total of nine motion directions for each block). Within each block, the order of the presentation of the nine motion directions was pseudo-randomized. In Figure 1D, we illustrated the nine directions for one of the coherent motion displays. The range of the nine motion directions covered the global motion direction (shown as 0°), the local motion directions (from −30° to +30° relative to the global motion direction) as well as other motion directions (from −60° to −30°, and from +30° to +60° relative to the global motion direction) outside the local motion directions. Two neighboring directions were 15° apart from each other. 

In one trial, subjects performed a motion discrimination task with 10% coherent motion (as shown in Figure 1C). At the beginning of each trial, a fixation point was presented for 400 ms, followed by the presentation of a coherent motion display with 10% coherence for 500 ms. Thereafter, a response screen with nine arrows and corresponding keyboard buttons, indicating the possible motion directions was shown. In the response screen, the range of nine arrows were fixed for each block. Subjects were asked to indicate which motion direction was presented by pressing the corresponding key on the keyboard. There was no time limit for responding and no feedback about response accuracy was provided. 

Sensitivity changes for each motion direction from pretest to posttest were calculated as d-prime, corresponding with z (hit rate) – z (false alarm rate). An increase in d-prime from pretest to posttest would be indicative of VPL. The tuning function of performance changes in dependence of the motion direction was calculated by fitting a smooth spline with piecewise polynomials to the mean d-prime across subjects at each motion direction. The smoothing parameter was 0.5. The mean sensitivity changes for trained and untrained local motion ranges were calculated by summing d-prime for each direction across the trained and untrained local motion ranges for each subject (including the global motion direction) and by averaging the results across subjects. 

fMRI motion decoder construction

The fMRI motion decoder construction stage was conducted inside the MRI scanner to obtain the blood-oxygen-level-dependent (BOLD) signal patterns in V1/V2 corresponding with the two Sekuler displays. 

Before the BOLD measurements a high-resolution anatomical brain scan was acquired with Siemens’ AutoAlign function (see MRI parameters) for each subject. By using the AutoAlign function, all subsequent functional scans were positioned automatically on the same slice of the brain, which was aligned to the anterior commissure to posterior commissure (AC-PC) plane. The high-resolution anatomical brain scan was reconstructed and the cortical surface inflated using the Freesurfer software package (Martinos Center for Biomedical Imaging, Charlestown, MA). 

Then, subjects’ BOLD signals were collected for the decoder construction runs while they were passively exposed to the two Sekuler displays. Subjects were instructed to maintain fixation on a white bull's eye on a gray disc while being exposed to 240 trials of the Sekuler display stimulus for a total of 10 runs. Each run had a duration of 300 seconds and consisted of 24 trials of Sekuler display stimuli (12 trials for each global motion direction). A 10-second fixation period and a 2-second fixation period were inserted at the beginning and end of each run, respectively. The initial fixation period was added to allow the magnetic field to reach a steady state. Each trial was 6 seconds long, during which one of the two global motion directions was presented. During stimulus presentation the color of the fixation point could change in an unpredictable fashion from white to green for 500 ms. The fixation point changed color in 50% of the trials (12 trials in one run). The order of trials with and without color change were randomly determined for each subject. Subjects were asked to press a button with their index finger if they noticed a color change. Each trial was followed by a 6-second-long fixation period in which the color of the fixation remains unchanged. 

Decoder construction runs were preprocessed using the FSFast toolbox contained within the FreeSurfer software package (Dale, Fischl, & Sereno, 1999; Fischl, 2012). Each run underwent three-dimensional motion correction to align all functional scans to the first scan collected during the first run of decoder construction. The average motion for each subject across different runs was less than the size of one voxel. Rigid body transformation was then performed to align the functional runs to each subject's high-resolution anatomical image. A gray matter mask was calculated for further analysis. No spatial smoothing or intensity normalization were performed. 

In the same session after decoder construction, retinotopic mapping runs were conducted with a standard procedure for retinotopy to delineate different visual areas for each subject (Engel et al., 1994; Fize et al., 2003; Yotsumoto, Watanabe, & Sasaki, 2008). The retinotopic stimulus consisted of a checkerboard in different colors extending from 1.00° to 4.35°. The retinotopic stimuli occupied a smaller region than the Sekuler display stimuli to avoid selecting voxels that corresponded with the edges of the stimuli. The checkerboard stimuli alternated between horizontal and vertical meridians as well as upper and lower visual fields. Subjects maintained central fixation and pressed a button when they noticed a change of color of the fixation point. For each frame of stimulus presentation, there was a 0.01 probability that the color of the fixation point changed from white to red. Retinotopic mapping runs were preprocessed using similar steps as for preprocessing of the decoder construction runs, but were additionally preprocessed with spatial smoothing and intensity normalization. The V1/V2 were delineated by computing the contrast between the horizontal and vertical meridians and projected onto the inflated cortical surface. 

Next, the time-course of BOLD activations from the decoder construction runs were extracted from voxels corresponding with V1/V2 using MATLAB. These functional data were then shifted by 6 seconds to account for the hemodynamic response delay. Linear trends were removed, followed by z-score normalization. The data samples used for decoding for each trial were the averaged intensities of three functional volumes that corresponded with the 6-second stimulus period of each run. Thus, we acquired 240 data samples that corresponded to 240 trials for each subject. 

Finally, we constructed the decoder for the two global motion directions corresponding with the two types of Sekuler displays using sparse logistic regression (Miyawaki et al., 2008; Yamashita, Sato, Yoshioka, Tong, & Kamitani, 2008). The input to the algorithm were the 240 data samples of voxels in V1/V2. Sparse logistic regression automatically selected voxels relevant for the separation of the representation of the two global motion directions. The output from the calculation was a decoder that consisted of estimated weights for each selected voxel in V1/V2. Leave-one-run-out cross-validation was conducted. For each round of the cross-validation, 216 data samples from nine runs were used to train the decoder with sparse logistic regression. The accuracy of the decoder was then tested using the 24 samples from the remaining run. The decoder was used to predict the likelihood of each global motion direction and assign one direction to each test sample. The accuracy of the decoder's prediction was then calculated by comparing the predicted direction with the true global motion direction that was presented. Thus, we acquired the test accuracy for each run in each cross-validation round for each subject, and the accuracy of the cross-validation was averaged across 10 cross-validation rounds for each subject. The statistical significance was determined with one-sample t tests. 

fMRI neurofeedback training

The decoder constructed during the decoder construction session was applied to the neurofeedback training sessions. The eighth volume image from the first decoder construction run was also obtained as a template image for real-time evaluation of the quality (e.g., motion artifact) of the functional runs (as discussed below in this article). One of the two Sekuler displays was pseudorandomly selected as the trained Sekuler display for each subject. Initially, a structural scan with AutoAlign was performed to automatically place the slice on the same plane as during the decoder construction session. 

Then, we conducted a number of induction runs in which subjects attempted to induce the trained Sekuler display pattern in V1/V2 with neurofeedback without realizing they were being trained (as discussed in the Motion stimulus section in this article). Each induction run lasted for 330 seconds, starting with a 30-second fixation period followed by fifteen 20-second-long trials, as shown in Figure 2. Subjects were instructed to maintain central fixation throughout the runs. Each trial started with a 6-second induction period. During the induction period, a “+” sign was presented at the center of the screen and subjects were instructed to try to use the posterior part of their brain to make a later presented green disc as large as possible. The induction period was followed by a 6-second-long fixation period, during which a “−” sign was presented at the center of the screen while subjects were instructed to fixate only. The fixation period was added to accommodate the 6-second hemodynamic response delay. The feedback period was presented afterwards for 2 seconds. A green disc was presented at the screen center. The size of the disc was proportional to the likelihood of the trained Sekuler display being represented by the fMRI activation pattern in V1/V2 during the induction period. The larger the size of the disc, the greater was the likelihood of the trained Sekuler display being represented by the fMRI activation patterns in V1/V2. There was a 5° gray boundary for the disc. Each trial ended with a 6-second intertrial interval while an “=” sign was shown. Subjects were instructed to relax but maintain their fixation during the = period. 

Figure 2.

 

One trial in an induction run. The “+” indicated the induction period, during which subjects tried to engage their brain activities to make a subsequently presented green disc larger. The “−” and “=” fixation and intertrial interval (ITI) periods, respectively. Feedback was given to subjects for 2 seconds by disk size. Subjects were instructed to maintain fixation throughout the run.

Each volume was transferred in real time from the scanner to a remote computer in which BOLD activations were calculated using MATLAB and SPM (Penny, Friston, Ashburner, Kiebel, & Nichols, 2011). The functional volumes that corresponded with the induction period were three-dimensional motion corrected in real time with SPM. The spatial correlation was calculated between the current volume and the template acquired from the decoder construction session. The correlation score was only shown to the experimenter, who was outside the scanner room. A correlation score of less than 0.85 indicated a significant movement of the subject in the scanner, which could lead to an inaccurate selection of voxels. The run was terminated if such a correlation score was observed, and a structural scan with AutoAlign was performed for accurate functional slice placement. Later, the BOLD activation time course was extracted from voxels corresponding with selected voxels for decoding in V1/V2. Then, linear trends were removed from the BOLD time course. After linear trend removal, z-score normalization was performed for each of the voxels to the 10- to 30-second functional volumes that were acquired from the onset of the run. Next, the intensities of the functional volumes that corresponded to the 6-second induction period were averaged to create the input data sample for sparse logistic regression. Finally, the likelihood of the current induction period was calculated using the data sample and the precalculated weights from the decoder construction stage. The likelihood ranged from 0% to 100% and was reflected proportionally by the size of the green disc. 

Subjects were allowed to take a break upon request. The mean ± standard deviation number of induction runs for each neurofeedback training session across subjects was 10.63 ± 1.5 runs. After completion of all neurofeedback training sessions and the posttests, subjects were asked to freely report which strategies they used and/or what they had in mind during each of the three neurofeedback training sessions. The results of their strategies are shown in Table 1. Importantly, none of the reported strategies by any subject were related to the imagination of visual motion. After the posttest, subjects were also asked to guess which motion range was trained during the neurofeedback training sessions, their responses were significantly less than the chance level (50%). 

Table 1.

 

Reported strategies during the neurofeedback training stage.

 

MRI parameters

All subjects were scanned in a 3 Tesla Siemens PRISMA MRI scanner using a 64-channel head coil. During each fMRI session a T1-weighted magnetization prepared rapid gradient echo sequence (256 slices, voxel size = 1 mm × 1 mm × 1 mm, TR = 1980 ms, TE = 3 ms, flip angle = 9°), was acquired with AAScout AutoAlign (160 slices, voxel size = 1.625 mm × 1.625 mm × 1.6 mm, 0 mm slice gap, TR = 3 ms, TE = 1.37 ms, flip angle = 8°) to ensure accurate placement of functional slices at the same location across sessions. For retinotopic mapping, decoder construction and induction scans, a T2*-weighted echoplanar imaging sequence was used including 33 continuous slices (TR = 2 s, TE = 30 ms, flip angle = 90°, voxel size = 3 mm × 3 mm × 3 mm) oriented parallel to the AC-PC plane placed by AutoAlign to cover the whole brain. 

Off-line leak analyses

We performed off-line leak analyses to test whether the target local motion-related activation patterns in V1/V2 leaked out of V1/V2 to other brain regions. Even if our fMRI neurofeedback training successfully induced the target motion-related activation patterns in V1/V2 and this resulted in sensitivity enhancements indicative of VPL, it would not unequivocally indicate that the behavioral changes are explained merely by the induced activation patterns in V1/V2. It is possible that, in parallel with the specific activation patterns in V1/V2, similar activation patterns occurred in other brain regions during the induction period of the neurofeedback training and contributed to the behavioral changes. To this end, we used the leak analysis, termed a region of interest (ROI)-based method as in previous studies (Shibata, Watanabe, Kawato, & Sasaki, 2016; Shibata et al., 2011). In the ROI-based method, we used anatomically delineated ROIs, including three motion-related regions and regions in frontal and parietal cortex in addition to V1/V2 and compared the amounts of leakage in these regions. The details of the leak analysis are described below. 

We used the ROIs of V1/V2 (the target area during neurofeedback training stage), V3A, the middle temporal area (MT), the medial superior temporal area (MST), the lateral intraparietal cortex (LIP), the intraparietal sulcus (IPS), and the frontal eye field (FEF) for this analysis. While V1/V2 was delineated by retinotopic mapping, other regions were defined by using the parcellation of the human cerebral cortex proposed by Glasser and colleagues (2016). After defining ROIs, we reconstructed the likelihood of the target motion-related activation patterns in V1/V2 obtained on-line (in real time) using a sparse linear regression algorithm (Toda, Imamizu, Kawato, & Sato, 2011) based on activation patterns measured in each of the aforementioned ROIs during the induction period of the neurofeedback training. Note that the likelihood we obtained on-line was the result of a nonlinear logistic function (0%–100%). Before we conducted the sparse linear regression, we transformed the likelihood of the target motion-related activation patterns with a hyperbolic tangent function. We trained the sparse linear regression (Shibata et al., 2016; Shibata et al., 2011) using data from two neurofeedback training sessions and then calculated the reconstructed value for the remaining neurofeedback training session with cross-validation in each ROI. Importantly, we trained the sparse linear regression in each ROI with the neurofeedback training sessions instead of training a decoder for each ROI with respect to the presented stimuli during the decoder construction stage. This current method has several benefits. First, it is not guaranteed that other ROIs could decode the Sekuler display during the decoder construction stage above chance level. Second, even if the decoding accuracy were above chance level, the decoders constructed in other ROIs might not decode the same content as the decoder in V1/V2. Therefore, such decoders would be less sensitive to detect the similarity between V1/V2 (the targeted region) activation patterns and activation patterns in other ROIs. By training the sparse linear regression with neurofeedback training sessions, we could evaluate how much brain activation patterns in other ROIs resembled the feedback scores, which reflected the brain activation patterns induced in V1/V2. We estimated the reconstruction performance of each ROI as a Fisher-transformed correlation coefficient between the reconstructed value and the likelihood of the target motion-related activation patterns in V1/V2 obtained on-line. After the estimation of the performance, we conducted permutation tests (see the Permutation test section) to evaluate the statistical significance of the estimated Fisher-transformed correlation coefficient for each ROI. 

Preprocessing for leak analysis

The functional runs from the neurofeedback stage were preprocessed with SPM (the same motion correction algorithm as in the real-time experiments) and MATLAB. All brain volumes underwent three-dimensional motion correction using the first run of the decoder construction stage as the template volume. A gray matter mask of the whole brain was calculated. The BOLD activation time course was extracted from the voxels corresponding with the whole brain gray matter mask. The time course was shifted by 6 seconds to account for the hemodynamic delay. Linear trends were removed, and z-score normalization was applied to the time course using the initial 10- to 30-second of the time course. For the neurofeedback induction runs, data samples for the leak analysis were acquired by averaging the 6-second induction period for each trial. 

Permutation test

The statistical significance of correlation coefficients calculated for the leak analysis was calculated using permutation testing. First, we permutated the order of the predicted probabilities from each control ROI 1,000 times and obtained 1,000 correlation coefficients between the reconstructed performance in the ROIs and the likelihood of the target motion in the activation patterns of V1/V2 acquired on-line. Second, we obtained the distribution of the correlation coefficients and tested whether the reconstructed performance acquired with the leak analysis was among the top 5% of the permutation distribution. A correlation coefficient was regarded as significant if it ranked among the top 5% of the distribution. Third, we computed the z-score for each control ROI by comparing the acquired correlation coefficient with the permutation distribution for between-subject comparisons. 

Control experiment

A behavioral control experiment consisted of identical stages as the neurofeedback experiment, except that no neurofeedback training was conducted and a break of the same duration as the neurofeedback training stage was included instead. All sessions of the control experiment were conducted outside the scanner (as shown in Figure 1E). 

The control experiment consisted of pretest and posttest sessions and a decoder exposure session. The procedures of the pretest and posttest sessnions were the same as in the neurofeedback experiment. Subjects were tested on 18 motion directions with 10% coherence. During the decoder exposure session, subjects were presented with the same Sekuler motion stimuli and fixation task as in the decoder construction session, except that the subjects were asked to respond to the fixation task by pressing a key on the keyboard in a psychophysical testing room instead of a press on the button box in the scanner. The posttest session was conducted at least 3 days apart from the decoder exposure session to ensure that the time courses of testing and training were as similar to those of the neurofeedback training sessions as possible. 

Code and data availability

All data and customized code for analyzing behavior and fMRI results are available upon request. 

Results

Neurofeedback experiment

The neurofeedback experiment consisted of four stages: the pretest stage, fMRI motion decoder construction stage, fMRI neurofeedback training stage and posttest stage. 

During the fMRI decoder construction session, a decoder was constructed to distinguish the BOLD activation patterns in V1/V2 corresponding with the two Sekuler display stimuli with different global motion directions and local motion ranges. Figure 3 shows that the mean decoding accuracy for the two Sekuler displays calculated using the leave-one-run-out cross-validation method was 57%, which was significantly above chance level (corresponding with 50%; p = 0.0049; Cohen's d = 1.43). This result indicates that the brain activation patterns in V1/V2 during real stimulus presentation distinguished between the trained and untrained local motion ranges in the two Sekuler display stimuli. 

Figure 3.

 

Decoding accuracy for trained versus untrained local motion ranges measured using a leave-one-run-out cross-validation method. The bar represents the mean decoding accuracy across subjects. The dashed line represents chance level for decoding accuracy (0.5 corresponding with a 50% chance level). Colored dots represent decoding accuracies for different subjects. The color assignment to each subject is the same as in Figure 5A, Figure 7, and Figure 8.

One of the two Sekuler display stimuli was selected as the trained pattern. During the fMRI neurofeedback training sessions, the subjects were instructed to induce the trained Sekuler motion pattern with an implicit feedback method (see Methods for details). The scores of neurofeedback training indicate the similarity between the subjects’ brain activation pattern in V1/V2 and the trained Sekuler motion pattern as determined in the decoder construction session. We measured the improvement in neurofeedback score by training as the change between the last and the first neurofeedback training session. The mean improvement of neurofeedback training scores across subjects was 5.41% ± 2.11 (standard error of the mean) and was significantly above zero, t(7) = 2.56; p = 0.038; Cohen's d = 0.905. This result indicates that subjects’ activation patterns in V1/V2 during the neurofeedback induction sessions became more similar to the trained Sekuler motion pattern over the course of neurofeedback training. 

During the pretest and posttest stages, subjects performed a motion direction discrimination task for nine motion directions surrounding each of the two global motion directions covering the local motion ranges in the two types of Sekuler display (see Figure 4). Each subject's performance change in the motion direction discrimination task was characterized as a change in d-prime (Posttest d-prime – pretest d-prime). A smooth spline curve was fitted for visualization purposes to the mean d-prime change across the trained local motion directions and the untrained local motion directions, separately, as shown in Figure 4. 

Figure 4.

 

Mean ± standard error of the mean d-prime changes across subjects in (A) the trained motion range and (B) the untrained motion range. On the x-axis, 0° represents the global motion direction. The y-axis shows changes in d-prime from pretest to posttest (d-prime in posttest minus d-prime in pretest). Values of greater than 0 (dashed line) indicate that the d-prime has improved from pretest to posttest. Values of less than 0 indicate that d-prime has decreased from pretest to posttest. The red line represents the spline fit across data points. The shaded area represents the local motion direction range.

As shown in Figure 5A, the d-prime change, summed across all local motion directions for each subject (see Methods for details), was significantly greater than zero, t(7) = 2.57; p = 0.037; Cohen's d = 0.91, indicating that subjects’ sensitivity for the trained local motion range has improved after neurofeedback training. In contrast, no significant change in d-prime was obtained in the untrained local motion range, t(7) = −0.866; p = 0.415; Cohen's d = −0.3062 (Figure 5A), indicating that the sensitivity for the untrained local motion range did not improve after neurofeedback training. There was no significant difference between the d-prime changes for the trained and untrained local motion ranges, t(7) = 1.719; p = 0.129; Cohen's d = 0.607 (Figure 5A). The improvement for both the trained global motion direction, t(7) = 0.9601; p = 0.369; Cohen's d = 0.339, and the untrained global motion direction, t(7) = 0.046; p = 0.965; Cohen's d = −0.016, was not significantly greater than zero (0° motion direction in Figure 4A and Figure 4B). As shown in Figure 5A, all subjects showed greater improvement in the trained local motion range, whereas no such tendency was observed for the untrained local motion range. 

Figure 5.

 

(A) Changes in sensitivity for trained and untrained local motion ranges corresponding with the average changes across trained and untrained directions, respectively (see Figure 4), across subjects (bar). Each color shows the result of a different subject. The color assignments to different subjects are the same as in Figure 3; otherwise they are the same as Figure 4. (B) Local motion range changes in the control experiment. Because there was no trained versus untrained local motion ranges, the motion ranges were specified as clockwise versus counterclockwise of vertical on the x-axis. Each same-colored dot represents the same individual subject's data; otherwise, they are the same as A.

We further explored the hit and false alarm rates for the trained local motion ranges, as shown in Figures 6A and B. The d-prime improvement (see Figure 4) was mainly driven by an increase in hit rate. Individual plots show a stronger tendency of hit rate improvements in local motion directions in the trained range (Figure 7A) than those of untrained local motion directions in the untrained range (Figure 7B). 

Figure 6.

 

Mean ± standard error of the mean changes in (A) hit rate and (B) false alarm rate for the trained motion range from pretest to posttest across subjects in the neurofeedback experiment. The y-axis shows changes in hit rate and false alarm rate from pretest to posttest (rate in posttest minus rate in pretest). Values of greater than 0 (dashed line) indicate that the rate has improved from pretest to posttest. Values of less than 0 indicate that the rate has decreased from pretest to posttest; otherwise, they are the same as Figure 4.

Figure 7.

 

Changes in hit rate for the (A) trained and (B) untrained motion ranges from pretest to posttest (posttest minus pretest) in the neurofeedback experiment. The thick black line shows the mean hit rate change across subjects after spline fitting. Each colored line shows the hit rate from a different subject after spline fitting. The color assignment is the same as in Figure 3; otherwise, they are the same as Figure 6.

ROI-based leak analysis

To check whether the behavioral performance changes were primarily correlated with activation patterns in V1/V2 or also with other visual areas or brain regions, we conducted an ROI-based leak analysis. We defined visual motion-sensitive brain regions (V1/V2, V3A, MT, and MST) and also other regions in the parietal and frontal cortex (LIP, IPS, and FEF) and measured the reconstructed probability of the trained visual motion patterns with sparse linear regression in each brain region during the neurofeedback training stage. If primarily V1/V2 were involved in driving the behavioral performance changes during the neurofeedback training, we would expect only V1/V2 to be capable of reconstructing the neurofeedback training scores. If, in contrast, other regions were also involved in the process, we would expect these regions to exhibit the capability of reconstructing the neurofeedback training scores. The reconstructed performance was defined as the Fisher-transformed correlation coefficient between the reconstructed probability and the estimated probability in V1/V2 from real-time experiments (see Methods for details). The reconstructed performance was significantly greater than zero for V1/V2 (z = 1.756; p = 0.039). In contrast, the correlation coefficients in none of the other regions were significantly higher than zero (MT: z = 0.582, p = 0.28; MST: z = 0.204, p = 0.419; V3A: z = −0.635, p = 0.262; LIP: z = 0.614, p = 0.269; IPS: z = −0.022, p = 0.491; FEF: z = 0.686, p = 0.246) (Figure 8). These results indicate that the behavioral performance changes in the neurofeedback experiment were primarily driven by neurofeedback-related training effects in V1/V2. 

Figure 8.

 

Mean Fisher-transformed correlation coefficient between the estimated scores of the V1/V2 activation patterns during the neurofeedback stage and the reconstructed scores of the activation patterns in V1/V2 (red) and in other control ROIs (blue). Only the activation pattern in V1/V2 (red) can significantly reconstruct the estimated scores of V1/V2 activation pattern during neurofeedback stage. Dots in the same color for different areas are from the same subject.

Control experiment

To rule out the possibility that the improvements in the local motion ranges in the neurofeedback experiment were induced by the repeated performance of the motion discrimination task in the pretest and posttest and/or by the exposure to the visual motion stimuli during the decoder construction stage, we conducted a control experiment (see Methods for details). During this control experiment, a new group of subjects performed the same task in the testing stages as in the neurofeedback experiment, except that no neurofeedback training was conducted. Because there was no trained versus untrained local motion ranges, we specified the two types of Sekuler motion display as clockwise or counterclockwise from the vertical direction. The results of the control experiment show no significant performance improvements in either the clockwise, t(7) = 1.172; p = 0.279, or counterclockwise local motion ranges, t(7) = 0.365; p = 0.725 (Figure 5B). Furthermore, subjects’ sensitivity changes from pretest to posttest for the trained local motion range in the neurofeedback experiment were significantly different from subjects’ sensitivity changes in the clockwise motion range, t(14) = 2.495; p = 0.025, and the counterclockwise motion range, t(14) = 2.198; p = 0.045, in the control experiment (Figure 5B). These results indicate that the sensitivity improvements for the trained local motion range in the neurofeedback experiment were unlikely related to test–retest effects or occurred because of exposure to visual motion during decoder construction. 

Discussion

A psychophysical study (Watanabe et al., 2002) found that repeated exposure to a Sekuler motion display improved the sensitivity in the local motion direction range but not specifically the global motion direction. The early phases of training on global motion discrimination in the Sekuler motion display led to performance improvements in the local motion direction ranges, whereas in later phases of training, performance improvements occurred in the global motion direction (Watanabe et al., 2002). These results indicated that exposure to a local visual feature leads to improvements in sensitivity for this feature, irrespective of whether the feature is task-relevant. However, these previous psychophysical results did not specify which brain areas are involved in this exposure-based task-irrelevant VPL. 

In the current study, we trained subjects with online decoded fMRI neurofeedback in which a decoded activation pattern related to the Sekuler display with both global and local motions was repetitively induced in V1/V2 while subjects were not presented with a Sekuler display and were not aware of the purpose of the experiment. The results showed a tendency for an increase in sensitivity for motion directions in the range in which local motion directions were trained with neurofeedback whereas no sensitivity changes occurred outside this range. In addition, the improvement was not specific to the global motion direction. The results of a leak analysis indicated that activation patterns in regions other than V1/V2 including higher-order visual motion areas and regions in frontoparietal cortex were not correlated with the target local motion patterns. These results are in accordance with the hypothesis that V1/V2 are involved in exposure-based task-irrelevant VPL, whereas the involvement of other brain regions is less likely. 

We believe that the VPL of local motions occurred without subjects’ awareness of the purpose of the experiment for the following reasons. First, subjects’ post-training verbal reports (Table 1) indicated that they were not aware of what was trained during neurofeedback. Subjects’ strategies during the neurofeedback training sessions did not involve anything related to the trained visual motion. Additionally, subjects were unable to guess their trained motion directions after the experiment (see the fMRI neurofeedback training section for details). Second, subjects were provided with no explicit cue to the purpose of experiment. Subjects were only asked to make the size of the presented disk as large as possible during the neurofeedback training. However, they were not informed about the underlying mechanics that made the disk size either larger or smaller. In addition, there was no cue in the procedure to lead subjects to notice which motion directions were supposed to be learned. For example, during test stages, subjects were exposed to each of the trained and untrained local motion directions with equal frequency. Although we cannot fully exclude the possibility that some (implicit) bias might have occurred because subjects completed the pretest and the decoder construction stages before the neurofeedback training and were thus exposed to visual motion patterns, we believe it is highly unlikely. 

One may argue that, if exposure-based learning occurs passively in V1/V2, we should have also observed sensitivity improvements for local motion after exposure to the Sekuler motion pattern for 1 hour in the control experiment. However, this is unlikely for the following reason. In Watanabe et al. (2002), the Sekuler motion pattern was exposed to subjects in each session for one hour over the course of 45 sessions total on separate days. Significant improvements were not observed until the tenth training session. This result indicates that a single, 1-hour-long exposure session to the Sekuler display is not sufficient to induce any changes in sensitivity for local motion. This result further indicates that the DecNef technique is a much more efficient training method than passive exposure to the training stimulus. 

The number of subjects used for the neurofeedback experiment of the present study was smaller than in some other studies conducted by our group (Amano et al., 2016; Shibata et al., 2011). Although statistically significant results were obtained with the support of individual tendencies, the same pattern of results with a larger number of subjects would allow us to draw stronger results and conclusions. For example, although the results of the neurofeedback experiment showed a tendency for general improvements within the trained local motion directions range, not all of the trained local motion directions exhibited such an improvement (e.g., the 0° local motion). We believe that a larger number of subjects would provide a clearer result for each local motion direction. 

Generally speaking, adequate control experiments are important to support conclusions drawn from the results of neurofeedback experiments (Paret et al., 2019; Shibata et al., 2019; Watanabe et al., 2017). In our study, to rule out any influences of testing effects on the results of our neurofeedback experiment, we conducted a control experiment in which only the pretest and posttest were conducted without neurofeedback training. The results of the control experiment ruled out the possibility that the results of the neurofeedback experiment were contaminated by testing effects. At the same time, conducting a control experiment in which sham or yoked neurofeedback were given during the training stage with otherwise identical procedures as in the neurofeedback experiment would certainly strengthen our conclusions. 

In summary, in the present study we trained subjects with decoded fMRI neurofeedback to induce activation patterns in V1/V2 that were similar to those evoked by a real presentation of a Sekuler display that consisted of random dots moving within a certain range of local motion directions, without actual presentation of a training stimulus. The results showed a tendency for an increase in sensitivity for the trained local motion directions but not particularly for the trained global motion direction. This tendency is in accordance with the hypothesis that early visual areas are involved in task-irrelevant VPL of visual motion, although a larger number of subjects in the neurofeedback experiment and a control experiment with yoked or fake feedback for future research would significantly strengthen our conclusions. 

Acknowledgments

ZW, MT, SMF, TY, YS, and TW and the study was supported by NIH R21EY028329, R01EY027841, R01EY019466, and United States - Israel Binational Science Foundation BSF2016058. Z.W. was also supported by T32MH115895. MK was partially supported by AMED (Japan, JP18dm0307008). KS was partially supported by JSPS Kakenhi (19H01041). MSW was supported by NIGMS-NIH P20GM103645. 

Author contributions: ZW, YS, and TW designed the study. ZW and MT performed the experiments. KS, SMF, MSW, and TY provided technical support for real-time neurofeedback. ZW analyzed the data. ZW, SMF, KS, TY, MK, YS, and TW wrote the manuscript. 

Commercial relationships: KS, MK, YS, and TW are the inventors of patents related to the neurofeedback method described in this article, and the original assignee of the patents is ATR, with which these authors are affiliated. 

Corresponding author: Takeo Watanabe 

Email: [e-mail]. 

Address: Brown University, Box 1821, 190 Thayer St, Providence, RI 02912, USA.