Evaluating the performance of the staircase and quick Change Detection methods in measuring perceptual learning

Abstract

The staircase method has been widely used in measuring perceptual learning. Recently, Zhao, Lesmes, and Lu (2017, 2019) developed the quick Change Detection (qCD) method and applied it to measure the trial-by-trial time course of dark adaptation. In the current study, we conducted two simulations to evaluate the performance of the 3-down/1-up staircase and qCD methods in measuring perceptual learning in a two-alternative forced-choice task. In Study 1, three observers with different time constants (40, 80, and 160 trials) of an exponential learning curve were simulated. Each simulated observer completed staircases with six step sizes (1%, 5%, 10%, 20%, 30%, and 60%) and a qCD procedure, each starting at five levels (+50%, +25%, 0, −25%, and −50% different from the true threshold in the first trial). We found the following results: Staircases with 1% and 5% step sizes failed to generate more than five reversals half of the time; and the bias and standard deviations of thresholds estimated from the post hoc segment-by-segment qCD analysis were much smaller than those from the staircase method with the other four step sizes. In Study 2, we simulated thresholds in the transfer phases with the same time constants and 50% transfer for each observer in Study 1. We found that the estimated transfer indexes from qCD showed smaller biases and standard deviations than those from the staircase method. In addition, rescoring the simulated data from the staircase method using the Bayesian estimation component of the qCD method resulted in much-improved estimates. We conclude that the qCD method characterizes the time course of perceptual learning and transfer more accurately, precisely, and efficiently than the staircase method, even with the optimal 10% step size.

Introduction

Perceptual learning refers to the improvement in an observer's performance in a perceptual task through training or practice. It has been documented in all sensory modalities (vision: Goldstone, 1998; Lu, Hua, Huang, Zhou, & Dosher, 2011; Sagi, 2011; Sasaki, Náñez, & Watanabe, 2010; Sasaki, Náñez, & Watanabe, 2012; Watanabe & Sasaki, 2015; hearing: Banai & Amitay, 2012; Moore, Amitay, & Hawkey, 2003; Wright & Zhang, 2009; smell: Stevenson, 2001; Wilson & Stevenson, 2003; taste: Blair & Hall, 2003; Mackintosh, Kaye, & Bennett, 1991; Scahill & Mackintosh, 2004; Symonds & Hall, 1995; touch: Rodríguez & Angulo, 2014; Sathian & Zangaladze, 1998). Perceptual learning not only unlocks important plasticity of the perceptual system (Petrov, Dosher, & Lu, 2005; Sagi, 2011; Sasaki et al., 2012) but also provides noninvasive rehabilitation methods for a variety of perceptual impairments such as amblyopia (C.-B. Huang, Zhou, & Lu, 2008; Polat, Ma-Naim, Belkin, & Sagi, 2004), myopia (Camilleri, Pavan, Ghin, & Campana, 2014; Tan & Fong, 2008; Yan et al., 2015), hemianopia (Nelles et al., 2001; Perez & Chokron, 2014), and aging (Andersen, Ni, Bower, & Watanabe, 2010; Bower & Andersen, 2012; Bower, Watanabe, & Andersen, 2013). 

One fundamental empirical phenomenon in perceptual learning is the learning curve. Most of the important discoveries about perceptual learning are based on the analysis of the learning curve in the training phase or the transfer phase. Indeed, the very existence of perceptual learning is based on whether the slope of the learning curve is significantly different from zero (Dosher & Lu, 2007; C.-B. Huang et al., 2008; Polat et al., 2004; Yan et al., 2015; Zhang, Hou, et al., 2018; Zhou et al., 2006). Changes of the learning curve obtained under different experimental manipulations such as external noise (Lu, Chu, & Dosher, 2006; Lu & Dosher, 2004), training difficulty (J. Liu, Lu, & Dosher, 2010, 2012; Z. Liu & Weinshall, 2000), training schedule (Hung & Seitz, 2014; Xiao et al., 2008), feedback (Aberg & Herzog, 2012; Fahle & Edelman, 1993; J. Liu et al., 2010, 2012; Shibata, Yamagishi, Ishii, & Kawato, 2009), attention (Donovan & Carrasco, 2015; Mukai et al., 2007; Szpiro & Carrasco, 2015), and reward (Seitz, Kim, & Watanabe, 2009; Zhang, Hou, et al., 2018) have revealed important properties of perceptual learning and are the bases for the development of theories and computational models of perceptual learning (Dosher & Lu, 2017). One of the hallmark properties of perceptual learning—specificity—is also derived from the learning curves during training and transfer (Ahissar & Hochstein, 1997; Fahle & Morgan, 1996; Z. Liu & Weinshall, 2000; Xiao et al., 2008). 

The learning curve in perceptual-learning studies is typically measured with the method of constant stimuli that estimates percentage correct (Fahle, Edelman, & Poggio, 1995; Fahle & Morgan, 1996), with d′ (Ball & Sekuler, 1982, 1987), or with adaptive procedures that estimate either contrast thresholds or difference thresholds in blocks of trials (T. Bi, Chen, Zhou, He, & Fang, 2014; Donovan & Carrasco, 2015; Liang, Zhou, Fahle, & Liu, 2015; Polat et al., 2004; Wang et al., 2016; Xiao et al., 2008) with various forms of the staircase procedure (Cornsweet, 1962; Watson & Pelli, 1983) as the most frequently used method (Leek, 2001; Meese, 1995; Monsen & Horn, 2007). 

The up-down staircase method is designed to estimate the threshold at a fixed performance level in a block of trials. Before each measurement, the initial stimulus level and the initial step size of the staircase are determined by the experimenter. The method adjusts the stimulus level and the step size based on the observer's performance and some predefined rules. For example, in a 3-down/1-up staircase procedure, the stimulus level is increased by one step if the observer makes one mistake (1 up) and is decreased by one step if the observer makes three consecutive correct responses (3 down). A reversal results when the direction of stimulus change is reversed (from up to down or down to up). In addition, the step size may decrease with the number of reversals. Typically, the staircase procedure requires a certain number of trials to generate enough reversals to provide a relatively precise estimate of the threshold. The performance of the staircase depends on the initial stimulus level and step size (Lu & Dosher, 2013). 

One major assumption underlying the staircase procedure is that the threshold does not change during the measurement block, which typically contains about 60–80 trials (Harris et al., 2015; Schoups, Vogels, & Orban, 1995; Sowden, Rose, & Davies, 2002; Thurston & Dobkins, 2007; Xiao et al., 2008). The staircase procedure has been widely used to measure time-varying phenomena such as adaptation (Bao, Fast, Mesik, & Engel, 2013; Binns, Taylor, Edwards, & Crabb, 2018; Jackson, Owsley, & McGwin, 1999; Ward, Morison, Simmers, & Shahani, 2018) and perceptual learning (Badiru, 1992; Heathcote, Brown, & Mewhort, 2000; Lu et al., 2011; Mazur & Hastie, 1978; Petrov et al., 2005). However, there has been no systematic study of the performance of the staircase procedure in measuring learning curves in perceptual learning, including the convergence properties of the procedure, its precision and bias, and the effects of the initial stimulus level and step size, in relation to different rates of learning. One goal of the current study is to systematically evaluate the performance of the staircase procedure in measuring perceptual learning and its transfer. 

Another goal of the current study is to compare the performance of the staircase procedure to that of a newly developed Bayesian adaptive procedure, the quick Change Detection (qCD) method, that has been shown to provide accurate and precise assessment of the dark-adaption curve (Zhao, Lesmes, & Lu, 2017, 2019) in an eight-alternative forced-choice (8AFC) task and perceptual learning in a four-alternative forced-choice (4AFC) task (Zhang, Zhao, Dosher, & Lu, 2018, 2019). Here we evaluate the performance of the qCD method in measuring the learning curves during both the learning and transfer phases of a two-alternative forced-choice (2AFC) experiment and compare it directly with that of the staircase procedure. This is quite important for two reasons: 2AFC is the most widely used procedure in perceptual-learning studies (a Google Scholar search on “perceptual learning and staircase” resulted in 3,380 links; Jogan & Stocker, 2014; Kingdom & Prins, 2010; Vancleef et al., 2018), and the number of alternatives in an mAFC procedure strongly affects the performance of the adaptive procedures (J. Bi, Lee, & O'Mahony, 2010; Hall, 1983; Hou, Lesmes, Bex, Dorr, & Lu, 2015; Shelton & Scarrow, 1984). Forced-choice procedures with three or four alternatives provide more satisfactory measurement of psychometric performance (Leek, 2001). Hou et al. (2015) found that increasing the number of alternatives in a forced-choice task greatly improved the efficiency of assessing a contrast sensitivity function in both simulation and psychophysical studies. Previously, we have used 4AFC and 8AFC tasks in comparing the staircase and qCD methods (Zhang, Zhao, et al., 2018, 2019; Zhao et al., 2017, 2019). 

The current article consists of two simulation studies. In Simulation Study 1, we evaluated the performance of the 3-down/1-up staircase and qCD methods in measuring the learning curves of three simulated observers with different learning rates. Five initial stimulus levels (+50%, +25%, 0, −25%, and −50% different from the true initial threshold) and six initial step sizes (1%, 5%, 10%, 20%, 30%, and 60% change) were included in the staircase procedure. Uniform priors on the parameters of the exponential curve were used in the qCD method. In Simulation Study 2, we conducted additional simulations to evaluate the accuracy and precision of the estimated transfer index from the qCD and staircase methods. 

Simulation Study 1

Methods

To evaluate the performance of the staircase and qCD methods in measuring the perceptual learning curve, we simulated three observers with exponential learning curves at three different rates of learning (Table 1) in a 2AFC task. 

Table 1

 

Parameters of the three simulated observers in Simulation Study 1.

 

The qCD method

In the qCD method (Figure 1; Appendix A), the true learning curve is characterized with an exponential function:  

\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\begin{equation}\tag{1}T\left( {\vec \theta ,n} \right) = \lambda {\rm{exp}}\left( {{{ - n} \over \gamma }} \right) + \alpha {\rm ,}\end{equation}

where Display Formula\({\vec {\theta}} = \left( {\lambda ,\gamma ,\alpha} \right)\) are the exponential parameters, Display Formula\(n\) is the trial number during training, and Display Formula\(T( {\vec \theta ,n} )\) is the threshold in trial Display Formula\(n\). In a 2AFC task, the probability-correct psychometric function is described by a Weibull function (Wichmann & Hill, 2001):  

\begin{equation}\tag{2a}{p^{\prime} _n}\left( {r = 1{\rm{|}}\vec \theta ,x} \right) = g + \left( {1 - g} \right)\left( {1 - {\rm{exp}}\left( { - {{\left( {{x \over {{T_w}\left( {\vec \theta ,n} \right)}}} \right)}^s}} \right)} \right){\rm ,}\end{equation}

 

\begin{equation}\tag{2b}{p_n}\left( {r = 1{\rm{|}}\vec \theta ,x} \right) = \left( {1 - l} \right)p_n^{\prime} \left( {r = 1{\rm{|}}\vec \theta ,x} \right) + lg{\rm ,}\end{equation}

where Display Formula\({p^{\prime} _n}\) is the psychometric function on trial n without a lapse rate and Display Formula\({p_n}\) is the psychometric function on trial n with a lapse rate, Display Formula\(r\) is the response (1 for correct, 0 for incorrect), Display Formula\(x\) is the stimulus value, and Display Formula\(g = 0.5\) is the guessing rate. The slope of the Weibull function was set as Display Formula\(s = 3.06\) and the lapse rate was set at Display Formula\(l = 0.04\).  

Figure 1

 

(A) An exponential function with three parameters: λ, the dynamic range of learning; γ, the time constant of the exponential function; and α, the asymptotic performance level. (B) The quick Change Detection method consists of five steps: (1) The learning curve—i.e., the time course of threshold—is defined as an exponential function (as a function of trial number n) with three parameters, θ = (α, γ, λ), and their joint prior distribution. (2) The stimulus level in the next trial is selected to optimize the expected information gain on the joint distribution of the parameters. (3) The posterior distribution is updated by Bayes's rule based on observer's response after each trial. (4) Steps 2 and 3 are repeated until the stop criterion is met (e.g., predetermined number of trials). (5) Trial-by-trial and post hoc segment-by-segment thresholds are computed from the posterior distributions.

A prior distribution Display Formula\({p_0}( {\vec \theta } )\) (see Appendix A for details) is used to represent a priori knowledge of the parameters of the learning curve at the beginning of the experiment. The qCD method selects the stimulus on each trial from all possible stimulus values to optimize the expected information gain (Kujala & Lukka, 2006; Lesmes, Lu, Baek, & Albright, 2010) and updates the joint posterior distribution based on the response of the observer via Bayes's rule. Stimulus selection and Bayesian updating are iterated until the total number of training trials is completed. Based on the joint posterior distribution after each trial, the qCD provides trial-by-trial estimates of the parameters of the perceptual-learning function and a single threshold estimate. In addition, a post hoc analysis can be performed on the trial-by-trial data to investigate the functional form of the learning curve, and this information can be used to further improve the accuracy and precision of the estimated thresholds by aggregating information across all trials (Zhao et al., 2017, 2019). 

To simulate the qCD method, a broad joint prior distribution Display Formula\({p_0}( {\vec \theta } )\) was defined in the three-dimensional parameter space of Display Formula\(\vec \theta = \left( {{\theta _1},{\theta _2},{\theta _3}} \right) = \left( {\lambda ,\gamma ,\alpha } \right)\), with the subscript 0 denoting that the prior represents a priori knowledge of the parameters of the learning curve before the experiment (Appendix A). The parameter space included 50 log-linearly spaced λ values (from 0.1 to 0.4), 50 log-linearly spaced γ values (from 20 to 200), and 50 log-linearly spaced α values (from 0.05 to 0.2). Both λ and α are in the units of the threshold measurement, while λ values are in the unit of trial number. The joint prior distribution Display Formula\({p_0}( {\vec \theta })\) was updated trial by trial throughout the simulated experiment. 

The 3-down/1-up staircase procedure

The 3-down/1-up staircase method (Cornsweet, 1962) was simulated with five starting stimulus levels (+50%, +25%, 0%, −25%, and −50% from the true initial threshold) and six step sizes (1%, 5%, 10%, 20%, 30%, and 60% of the current stimulus level). We denote the staircases with their step sizes—for example, SC10% represents a staircase with a 10% step size. In the 3-down/1-up staircase procedure, if the simulated observer makes three consecutive correct responses, the stimulus level (e.g., contrast, luminance, orientation) is reduced by the step size (multiplied by 1 − step size) in the next trial (3 down). If the simulated observer makes a single incorrect response, the stimulus level is increased by the step size (multiplied by 1 + step size) in the next trial (1 up). A reversal (or endpoint) happens if the staircase changes its direction (from upward to downward or vice versa). Usually, one block of 3-down/1-up staircase with a 10% step size produces about a dozen reversals in 80 trials. The estimated threshold of the block is calculated by averaging stimulus levels at even numbers of reversals after deleting the first four or five of them. 

Simulation procedures

We simulated 1,000 runs of 800 trials each for each observer with each of the 35 measurement procedures—qCD and the 3-down/1-up staircase with six step sizes, each starting at one of five starting stimulus levels—resulting in 28,000,000 simulated trials per observer. Each measurement block of the staircase method consisted of 40, 80, or 160 trials. In order to investigate the effects of the initial stimulus level on its performance, we did not use optimal stimulus selection in the first trial of the qCD procedure. Instead, we matched the initial stimulus level in the qCD and staircase procedures. 

In each trial, the true threshold T(n) was calculated using Equation 1. The expected probability of a correct response was calculated using Equation 2b. To determine whether the observer's response was correct, we first drew a random number Display Formula\(\rho \) from a uniform distribution over the interval from 0 to 1 and then labeled the response as correct if Display Formula\(\rho \) < Display Formula\({p_n}( {r = 1{\rm{|}}\vec \theta ,x} )\), and incorrect otherwise. 

Evaluation metrics

Accuracy is a measure of the difference between the estimated and true values, usually computed as the bias of the estimates. Precision is gauged by the variability of the estimates (Lu & Dosher, 2013; Treutwein, 1995). We computed the bias and variability of the estimated thresholds and parameters of the learning curves. Logarithmic-scale (log₁₀ unit) measures were used in the following analyses because this allowed us to compare the quality of measures in different dimensions. 

Accuracy

For the qCD method, the bias (on a log₁₀ scale) of the estimated threshold of the nth simulated trial is defined as  

\begin{equation}\tag{3}Bia{s_n} = {{\mathop \sum \nolimits_{j} \left( {lo{g_{10}}\left( {{T_{nj}}} \right) - lo{g_{10}}\left( {{T_{n,true}}} \right)} \right)} \over J}{\rm ,}\end{equation}

where Display Formula\({T_{nj}}\) is the estimated threshold of the nth trial in the jth simulation and Display Formula\({T_{n,true}}\) is the true threshold in the nth trial.  

To compute a summary statistic of the bias over the entire learning curve, we cannot simply compute the mean of Display Formula\(Bia{s_n}\), because biases at different time points may carry opposite signs. Instead, we use the root mean square error (RMSE) to represent the average bias over the entire learning curve:  

\begin{equation}\tag{4}RMSE = \sqrt {{{\mathop \sum \nolimits_n {{\left( {lo{g_{10}}\left( {{{\bar T}_n}} \right) - lo{g_{10}}\left( {{T_{n,true}}} \right)} \right)}^2}} \over N}} {\rm ,}\end{equation}

 

\begin{equation}\tag{5}{\bar T_n} = {{\mathop \sum \nolimits_j {T_{nj}}} \over J}{\rm ,}\end{equation}

where N is the total number of trials in each run.  

The staircase method can only provide a single threshold estimate in each measurement block. Bias (on a log₁₀ scale) of the estimated threshold of the bth simulated block is defined as  

\begin{equation}\tag{6}Bia{s_b} = {{\mathop \sum \nolimits_j \left( {lo{g_{10}}\left( {{T_{bj}}} \right) - lo{g_{10}}\left( {{T_{b,true}}} \right)} \right)} \over J},\end{equation}

where Display Formula\({T_{bj}}\) is the estimated threshold in the bth block of the jth simulation, Display Formula\(J\) is the total number of simulations, and Display Formula\({\rm{\ }}{T_{b,true}}\) is the true threshold in the middle of the bth block—that is, the threshold in trial Display Formula\(\left( {b - 0.5} \right) \times blocksize\) on the simulated learning curve.  

The RMSE of the estimated thresholds of the entire learning curve from the staircase method is defined as  

\begin{equation}\tag{7}RMSE = \sqrt {{{\mathop \sum \nolimits_b {{\left( {lo{g_{10}}\left( {{{\bar T}_b}} \right) - lo{g_{10}}\left( {{T_{b,true}}} \right)} \right)}^2}} \over B}} {\rm ,}\end{equation}

 

\begin{equation}\tag{8}{\bar T_b} = {{\mathop \sum \nolimits_j {T_{bj}}} \over J}{\rm ,}\end{equation}

where B is the total number of blocks.  

The biases of the estimated parameters Display Formula\({\theta _f} = \left( {\lambda ,\gamma ,\alpha } \right)\) from the qCD method are defined as  

\begin{equation}\tag{9}Bia{s_{\left( {n,{\theta _f}} \right)}} = {{\mathop \sum \nolimits_j \left( {lo{g_{10}}\left( {{\theta _{f,nj}}} \right) - lo{g_{10}}\left( {{\theta _{f,true}}} \right)} \right)} \over J}{\rm ,}\end{equation}

where Display Formula\({\theta _{f,nj}}\) is the estimated parameter value after the nth trial in the jth simulation, Display Formula\(J\) is the total number of simulations, Display Formula\({\theta _{f,true}}\) is the true parameter value, and f = 1, 2, 3.  

For the staircase method, the estimated parameters were obtained from the best-fitting exponential function to the estimated block-by-block thresholds in each run using the MATLAB (MathWorks, Natick, MA) function fminsearch. The biases of the estimated parameters from the staircase method are defined as  

\begin{equation}\tag{10}Bia{s_{\left( {{\theta _f}} \right)}} = {{\mathop \sum \nolimits_j \left( {lo{g_{10}}\left( {{\theta _{f,j}}} \right) - lo{g_{10}}\left( {{\theta _{f,true}}} \right)} \right)} \over J}{\rm .}\end{equation}

 

Precision

We evaluated the precision of the procedures in two different ways. The first estimate is the standard deviation (SD) of repeated measures. For the qCD method, the SD of the estimated threshold of the nth simulated trial is defined as  

\begin{equation}\tag{11}S{D_n} = \sqrt {{{\mathop \sum \nolimits_j {{\left( {lo{g_{10}}\left( {{T_{nj}}} \right) - lo{g_{10}}\left( {{{\bar T}_n}} \right)} \right)}^2}} \over J}} {\rm .}\end{equation}

 

The average SD of the estimated threshold of the entire learning curve is defined as  

\begin{equation}\tag{12}{\overline {SD} _n} = \sqrt {{{\mathop \sum \nolimits_n {{\left( {S{D_n}} \right)}^2}} \over N}} {\rm .}\end{equation}

 

The staircase method can only provide block-by-block threshold estimates. The SD of the estimated thresholds in the bth block is defined as  

\begin{equation}\tag{13}S{D_b} = \sqrt {{{\mathop \sum \nolimits_j {{\left( {lo{g_{10}}\left( {{T_{bj}}} \right) - lo{g_{10}}\left( {{{\bar T}_b}} \right)} \right)}^2}} \over J}} {\rm ,}\end{equation}

 

\begin{equation}\tag{14}{\bar T_b} = {{\mathop \sum \nolimits_j {T_{bj}}} \over J}{\rm .}\end{equation}

 

The average SD of the estimated threshold of the entire learning curve from the staircase method is defined as  

\begin{equation}\tag{15}{\overline {SD} _b} = \sqrt {{{\mathop \sum \nolimits_b {{\left( {S{D_b}} \right)}^2}} \over B}} {\rm ,}\end{equation}

where B is the total number of blocks in each run.  

Similarly, the SDs of the estimated parameters from the qCD method are defined as  

\begin{equation}\tag{16}S{D_{\left( {n,{\theta _f}} \right)}} = \sqrt {{{\mathop \sum \nolimits_j {{\left( {lo{g_{10}}\left( {{\theta _{f,nj}}} \right) - lo{g_{10}}\left( {{{\bar \theta }_{f,n}}} \right)} \right)}^2}} \over J}} {\rm ,}\end{equation}

 

\begin{equation}\tag{17}{\bar \theta _f} = {{\mathop \sum \nolimits_j {\theta _{fj}}} \over J}{\rm .}\end{equation}

 

The SDs of the estimated parameters from the staircase method are defined as  

\begin{equation}\tag{18}S{D_{\left( {{\theta _f}} \right)}} = \sqrt {{{\mathop \sum \nolimits_j {{\left( {lo{g_{10}}\left( {{\theta _{f,j}}} \right) - lo{g_{10}}\left( {{{\bar \theta }_f}} \right)} \right)}^2}} \over J}} {\rm .}\end{equation}

 

The second way to gauge precision is by the half width of the 68.2% credible interval (HWCI) of the posterior distributions (Edwards, Lindman, & Savage, 1963). The HWCI can be used to assess the precision of a single run of the qCD procedure (Greenland & Kenneth, 1997). For example, the 68.2% credible interval approximates the range within which the actual value lies with 68.2% probability. This is especially important for perceptual-learning studies, because the learning curve can only be measured once in an individual; and therefore estimating SD from repeated experimental runs is not possible. 

Results

We tested the qCD and staircase methods with five starting stimulus levels: +50%, +25%, 0%, −25%, and −50% from the true threshold in the first trial. Because the patterns of results with the five starting levels are quite similar, we mainly present the results with the 0% starting level in the main text. The results with the other four starting levels are presented in the supplementary information. 

Staircase convergence

In the 3-down/1-up staircase procedure, the estimated threshold in each measurement block is calculated by averaging stimulus levels at an even number of reversals after deleting the first four or five of them. We computed the block thresholds from the endpoints in every 40, 80, or 160 trials. However, because many staircases with a block size of 40 did not generate more than five reversals (see Supplementary Table S1) and staircases with a block size of 160 produced an insufficient number of block thresholds (see Supplementary Figure S1), we only evaluated the performance of the staircase method with a block size of 80 trials in the following analyses. Table 2 shows the percentage of staircases with more than five reversals in the first measurement block of the learning curves. With step sizes of 1% and 5%, many staircases didn't generate more than five reversals. We therefore dropped these two conditions in subsequent analyses. In summary, we evaluated the performance of the staircase method only with four step sizes (10%, 20%, 30%, and 60%) and one block size (80 trials). 

Table 2

 

Percentage of staircase runs with more than five reversals in the first block of learning.

 

Estimated learning curves

The estimated learning curves in the first 300 trials in the trial-by-trial (red line) and post hoc segment-by-segment (blue line) analyses from the qCD method, and the block-by-block threshold estimates from the staircase method with a 0% starting level, are shown in Figure 2. Visual inspection suggests that the estimated post hoc segment-by-segment thresholds from the qCD method were very close to the true thresholds, and closer than the block-by-block thresholds from the staircase method (SC10%, SC20%, SC30%, and SC60%). In addition, the estimated thresholds from the qCD method were more precise than those from the staircase method. We quantify these observations next. The estimated thresholds with ±25% and ±50% starting levels are shown in Supplementary Figures S2 through S5. 

Figure 2

 

Estimated learning curves of the three simulated observers—(A) Observer 1, (B) Observer 2, (C) Observer 3—with a starting level at 0% from the true threshold in the first trial. Results of the first 300 trials from both the quick Change Detection and staircase methods are shown. Red and blue lines denote the estimates from trial-by-trial and post hoc segment-by-segment analyses of quick Change Detection simulation, respectively. Green circles, magenta downward-pointing triangles, cyan squares, and brown stars represent the block-by-block thresholds from the staircase method with step sizes of 10%, 20%, 30%, and 60%, respectively. The shaded area and error bars denote the standard deviation in the quick Change Detection and staircase methods, respectively. Black dash-dotted lines denote the true learning curves.

Accuracy and precision of the estimated thresholds

The biases of the estimated thresholds from the qCD and staircase methods with the 0% starting level are plotted in Figure 3A. The RMSEs of the estimated thresholds from the qCD trial-by-trial, qCD post hoc segment-by-segment, SC10%, SC20%, SC30%, and SC60% were 0.015, 0.005, 0.011, 0.014, 0.028, and 0.090 log₁₀ units, respectively, for Observer 1; 0.015, 0.004, 0.007, 0.015, 0.030, and 0.092 log₁₀ units for Observer 2; and 0.018, 0.002, 0.003, 0.015, 0.030, and 0.093 log₁₀ units for Observer 3. The qCD method was much more accurate than the staircase method. The RMSEs of the estimated thresholds with starting levels of ±25% and ±50% are given in Table 3; more details are provided in Supplementary Figure S6. 

Figure 3

 

Comparison of the accuracy and precision of the estimated thresholds from the quick Change Detection and staircase methods. The biases (A), standard deviations (B), and half widths of 68.2% credible intervals (C) of the three simulated observers with a starting level of 0% from the true threshold in the first trial are shown in separate rows as functions of trial numbers. Red and blue lines denote the results from the trial-by-trial and post hoc analyses of quick Change Detection simulation, respectively. Green circles, magenta downward-pointing triangles, cyan squares, and brown stars represent the results from the staircase method with step sizes of 10%, 20%, 30%, and 60%, respectively.

Table 3

 

Root mean square error of the estimated thresholds from the quick Change Detection (qCD) and staircase methods. Notes: qCDtrial = trial-by-trial qCD; qCDseg = post hoc segment-by-segment qCD.

 

The SDs of the estimated thresholds from the qCD and staircase methods with the 0% starting level are shown in Figure 3B. The SDs of the estimated thresholds from the qCD trial-by-trial, qCD post hoc segment-by-segment, SC10%, SC20%, SC30%, and SC60% were 0.029, 0.017, 0.045, 0.052, 0.059, and 0.084 log₁₀ units, respectively, for Observer 1; 0.032, 0.018, 0.044, 0.052, 0.059, and 0.085 log₁₀ units for Observer 2; and 0.032, 0.017, 0.043, 0.052, 0.059, and 0.085 log₁₀ units for Observer 3. For the staircase methods, the SD increased with step size. The SDs of the estimated thresholds from the qCD method were always considerably smaller than those from the staircase methods. In summary, the precision of the estimated thresholds from the qCD method was much higher than that of those from the staircase methods. The SDs of the estimated thresholds with five starting levels are listed in Table 4; more details are provided in Supplementary Figure S7. 

Table 4

 

Standard deviations of the estimated thresholds in the quick Change Detection (qCD) and staircase methods. Notes: qCDtrial = trial-by-trial qCD; qCDseg = post hoc segment-by-segment qCD.

 

The 68.2% HWCIs of the estimated thresholds from the qCD method with the 0% starting level are shown in Figure 3C. Averaged across the three simulated observers, the 68.2% HWCIs of the estimated trial-by-trial threshold were 0.049, 0.042, 0.030, and 0.016 log₁₀ units, respectively, after 100, 200, 400, and 800 trials; for the post hoc segment-by-segment analysis across the three simulated observers they were 0.025, 0.018, 0.013, and 0.016 log₁₀ units. The 68.2% HWCIs with all five starting stimulus levels are listed in Table 5; more details are provided in Supplementary Figure S8. 

Table 5

 

Half widths of 68.2% credible intervals of the estimated thresholds (in log₁₀ units) from the quick Change Detection (qCD) method: trial by trial and post hoc segment by segment.

 

Accuracy and precision of the estimated parameters

Since SC20%, SC30%, and SC60% generated estimated thresholds with much lower accuracy and precision than SC10%, we compared only SC10% with the qCD method in the following analyses. The biases and SDs of the estimated parameters from the qCD and staircase methods with 0% starting levels are plotted in Figure 4. For the qCD method, the bias was computed from the post hoc segment-by-segment analysis. For the staircase method, we calculated the bias and SD of the best-fitting parameters of the exponential model to the estimated learning curves. The biases of the estimated parameters from the qCD method were smaller than those from the staircase method, especially for simulated Observer 1, who has the fastest learning rate among the three observers. 

Figure 4

 

The biases (A) and standard deviations (B) of the estimated parameters (λ, γ, and α) of the three simulated observers from the post hoc segment-by-segment quick Change Detection (blue) and SC10% staircase (green) methods.

For Observer 1, with the starting stimulus level at 0%, the biases of the estimated λ, Display Formula\(\gamma ,\) and Display Formula\(\alpha \) were 0.030, 0.028, and −0.001 log₁₀ units, respectively, from the qCD method and 0.149, −0.029, and −0.004 log₁₀ units from the staircase method; the SDs of the estimated λ, Display Formula\(\gamma ,\) and Display Formula\(\alpha \) were 0.074, 0.083, and 0.013 log₁₀ units from the qCD method and 0.489, 0.213, and 0.017 log₁₀ units from the staircase method. For Observers 2 and 3, with slower learning, the biases and SDs of the estimated parameters from the staircases were not as large as for Observer 1. However, qCD still provided more accurate (less biased) and precise estimates. In addition, different starting stimulus levels affected the accuracy of the estimated parameters from the staircase method but not the qCD method (see Supplementary Figure S9 for details). 

The 68.2% HWCIs of the estimated parameters from the qCD method are shown in Figure 5; they decreased with trial number. Averaged across the three observers, the 68.2% HWCIs of the estimated λ were 0.111, 0.087, 0.072, and 0.069 log₁₀ units, respectively, after 100, 200, 400, and 800 trials; for the estimated γ they were 0.196, 0.158, 0.105, and 0.074 log₁₀ units; and for the estimated α they were 0.169, 0.117, 0.051, and 0.018 log₁₀ units (see Supplementary Figure S10 for details on ±25% and ±50% starting levels). The results indicate that the qCD method can estimate the parameters of the learning curve with high precision. 

Figure 5

 

Half widths of 68.2% credible intervals of the estimated λ (A), γ (B), and α (C) of the three simulated observers with starting levels at 0% from the true threshold in the first trial in the trial-by-trial quick Change Detection simulation. Red, green, and blue dash-dotted lines denote Observers 1, 2, and 3, respectively.

Estimated initial threshold and percent of improvements

Accurate and precise estimates of the initial threshold (IT = λ + α) and percentage of improvements (PI = [λ + α]/α) are critical for understanding characteristics of perceptual learning, such as specificity, transfer, and retention. As shown in Figure 6A, the histogram of the estimated ITs from the qCD method are distributed symmetrically and tightly around the true IT (= 0.358). However, the histograms of the estimated IT from the staircase method have long tails in one direction, indicating systematic and sometimes large biases, with corresponding effects on the SD. For example, when the time constant was 40 trials (Observer 1), the estimated IT was 0.344 ± 0.041 (M ±SD) from the qCD method and 0.847 ± 1.608 from SC10%. 

Figure 6

 

Distributions of the initial threshold (A) and percentage of improvements (B) from the quick Change Detection and SC10% staircase methods with the 0% starting level. Blue asterisks and green circles denote results from the post hoc segment-by-segment quick Change Detection and staircase methods, respectively. Black dash-dotted lines denote the true values. Results from the three simulated observers are shown in separate rows.

The distributions of the estimated PI (Figure 6B) from the qCD method are also much narrower and closer to the true PI (= 400%) compared to those from SC10%. For example, when the time constant was 40 trials (Observer 1), the estimated PI was 384% ± 45% from the qCD method and 944% ± 1769% from SC10%. Obviously, the staircase method produced a less accurate mean and much larger SD for the estimated PI. These results demonstrate that the accuracy and precision of the estimated IT and PI from the qCD method were much higher than those from the staircase method. The means and SDs of the estimated IT and PI of the three observers with five starting levels are summarized in Table 6. 

Table 6

 

Means and standard deviations of initial threshold (IT) and percentage of improvements (PI) in the quick Change Detection (qCD) and staircase methods. Notes: qCDseg = post hoc segment-by-segment qCD.

 

Simulation Study 2

Method

Although Simulation Study 1 found that the estimated initial threshold, which is central to the formula for the transfer index (Equation 19), was more accurate and precise from the qCD method than from the staircase method, a direct evaluation of the methods in measuring transfer/specificity is still necessary. Therefore, to further evaluate the performance of the qCD and staircase methods in assessing the transfer index in a 2AFC task, we extended the simulations in Simulation Study 1 by adding a transfer phase to each of the simulated observers. Specifically, each simulated observer kept the same time constant and asymptotic performance level but had only half threshold reduction in the transfer phase (Table 7). 

Table 7

 

Parameters of the three simulated observers in the training and transfer phases of Simulation Study 2. Notes: TI = transfer index.

 

We conducted 1,000 simulated transfer runs for each observer with each method. Each run consisted of 800 trials of the qCD method and 800 trials of the 3-down/1-up staircase method (Cornsweet, 1962) with a 10% step size and 80-trial block size. Both methods were simulated with five starting stimulus levels (+50%, +25%, 0%, −25%, and −50% from the true initial threshold in the training phase). We used the same starting stimulus levels in the training and transfer phases (Donovan & Carrasco, 2015; Jeter, Dosher, Petrov, & Lu, 2009; Liang et al., 2015; Xiao et al., 2008; Zhang, Hou, et al., 2018). The qCD and staircase procedures were run separately and independently, and thus never shared any information. In other words, the terminology “training” and “transfer” in the current study is purely nominal. The detailed settings of Simulation Study 2 were the same as those of Simulation Study 1. 

The transfer index (TI) is defined as  

\begin{equation}\tag{19}{\rm{TI}} = 1 - {{{T_{B1}} - {T_{Bend}}} \over {{T_{A1}} - {T_{Aend}}}},\end{equation}

where Display Formula\({T_{A1}}\) and Display Formula\({T_{Aend}}\) are the estimated thresholds in the first and last trials of the training, and Display Formula\({T_{B1}}\) and Display Formula\({T_{Bend}}\) are the estimated thresholds for the first and last trials of the transfer. For the qCD method, TI was computed using the estimated thresholds from the post hoc segment-by-segment analysis. For the staircase method, the TIs were obtained from the best-fitting exponential function to the estimated block-by-block thresholds in each run. As shown in Table 7, the true TI of all three simulated observers was 50%.  

Results

Staircase convergence

Again, the estimated threshold in each measurement block of the staircase was calculated by averaging stimulus levels at an even number of reversals after deleting the first four or five of them. Table 8 shows that the percentage of staircases in the first block of the transfer that had more than five reversals was 100% in all but two conditions for Observer 1. 

Table 8

 

Percentage of staircases with more than five reversals in the first block of learning, by starting level.

 

Estimated learning curves in the transfer phase

For brevity, in the main text we present only the results from the qCD method and the staircase methods in the transfer phases using the 0% starting level. Visual inspection suggests that the estimated post hoc segment-by-segment thresholds from the qCD method were very close to the true thresholds, and closer than the block-by-block thresholds from the staircase method (Figure 7). In addition, the estimated thresholds from the qCD method were more precise (less variable) than those from the staircase method. We quantify these observations next. The estimated thresholds with ±25% and ±50% starting levels are shown in Supplementary Figures S11 through S13. 

Figure 7

 

Estimated learning curves in the transfer phase. Results from the first 300 trials of the three simulated observers are shown in separate rows. Red and blue lines denote the estimates from trial-by-trial and post hoc segment-by-segment analyses of quick Change Detection simulation, respectively. Green circles represent the block-by-block threshold from the SC10% staircase method. The shaded areas and error bars denote the standard deviations in the quick Change Detection and staircase methods, respectively. Black dash-dotted lines represent the true learning curves.

Accuracy and precision of the estimated thresholds in the transfer phase

The biases of the estimated thresholds from the qCD and staircase methods with the 0% starting level in the transfer phase are plotted in Figure 8A. The RMSEs of the estimated thresholds from the qCD trial-by-trial, qCD post hoc segment-by-segment, and SC10% were 0.013, 0.003, and 0.006 log₁₀ units, respectively, for Observer 1; 0.009, 0.003, and 0.005 log₁₀ units for Observer 2; and 0.015, 0.005, and 0.004 log₁₀ units for Observer 3. The qCD method was much more accurate than the staircase method when the learning rate was more rapid. The RMSEs of the estimated thresholds with starting levels of ±25% and ±50% are listed in Supplementary Table S2; more details are provided in Supplementary Figure S14. 

Figure 8

 

Comparison of the accuracy and precision of the estimated thresholds in the transfer phase. The biases (A), standard deviations (B), and half widths of 68.2% credible intervals (C) of the three simulated observers with starting level at 0% from the true threshold in the first trial are shown in separate rows as functions of trial numbers. Red and blue lines denote the results from the trial-by-trial and post hoc analyses of quick Change Detection simulations, respectively. Green circles represent the results from the SC10% staircase method.

The SDs of the estimated thresholds from the qCD and staircase methods with the 0% starting level in the transfer phase are shown in Figure 8B. The SDs of the estimated thresholds from the qCD trial-by-trial, qCD post hoc segment-by-segment, and SC10% were 0.027, 0.016, and 0.045 log₁₀ units, respectively, for Observer 1; 0.030, 0.017, and 0.044 log₁₀ units for Observer 2; and 0.031, 0.018, and 0.044 log₁₀ units for Observer 3. For the staircase methods, the SD increased with step size. The SDs of the estimated thresholds from the qCD method were always considerably smaller than those from the staircase methods. In summary, the precision of the estimated thresholds from the qCD method was much higher than from the staircase methods. More details are provided in Supplementary Table S3 and Supplementary Figure S15. 

The 68.2% HWCIs of the estimated thresholds from the qCD method with the 0% starting level in the transfer phase are shown in Figure 8C. Averaged across the three simulated observers, the 68.2% HWCIs of the estimated trial-by-trial threshold were 0.048, 0.040, 0.028, and 0.015 log₁₀ units, respectively, after 100, 200, 400, and 800 trials; from the post hoc segment-by-segment analysis they were 0.024, 0.017, 0.013, and 0.015 log₁₀ units. More details are provided in Supplementary Figure S16. 

Accuracy and precision of the estimated parameters in the transfer phase

The biases and SDs of the estimated parameters from the qCD and staircase methods with 0% starting level (relative to the true threshold in the beginning of the training phase) in the transfer phase are plotted in Figure 9. The biases of the estimated parameters from the qCD method were smaller than those from the staircase method, especially for simulated Observer 1, who has the fastest learning rate among the three observers. For Observer 1, with the starting stimulus level at 0%, the biases of the estimated λ, Display Formula\(\gamma ,\) and Display Formula\(\alpha \) were 0.042, −0.009, and −0.001 log₁₀ units, respectively, from the qCD method and 0.302, −0.042, and −0.028 log₁₀ units from the staircase method; the SDs were 0.087, 0.107, and 0.013 log₁₀ units from the qCD method and 0.978, 0.453, and 0.470 log₁₀ units from the staircase methods. For Observers 2 and 3, with slower learning, the biases and SDs of the estimated parameters from the staircases were not as large as for Observer 1. However, qCD still provided more accurate and precise estimates. In addition, different starting stimulus levels affected the accuracy of the estimated parameters from the staircase method but not the qCD method (see Supplementary Figure S17 for details). 

Figure 9

 

The biases (A) and standard deviations (B) of the estimated parameters (λ, γ, and α) in the transfer phase. Results from the post hoc segment-by-segment quick Change Detection analyses are shown in blue. Results from the SC10% staircase are shown in green.

The 68.2% HWCIs of the estimated parameters from the qCD method in the transfer phase are shown in Figure 10; they decreased with trial number. Averaged across the three observers, the 68.2% HWCIs of the estimated λ were 0.125, 0.109, 0.099, and 0.095 log₁₀ units, respectively, after 100, 200, 400, and 800 trials; for the estimated γ they were 0.276, 0.242, 0.150, and 0.109 log₁₀ units; and for the estimated α they were 0.151, 0.098, 0.040, and 0.017 log₁₀ units (see Supplementary Figure S18 for details on ±25% and ±50% starting levels). The results indicate that the qCD method can estimate the parameters of the learning curve with high precision. 

Figure 10

 

Half widths of 68.2% credible intervals of the estimated λ (A), γ (B), and α (C) from the quick Change Detection method in the transfer phase. Red, green, and blue dash-dotted lines denote Observers 1, 2, and 3, respectively.

Estimated IT and PI

The histogram of the estimated ITs and PIs from the qCD method is distributed symmetrically and tightly around the true values (IT = 0.224, PI = 250%), but that from the staircase method is not (see Figure 11), indicating that the qCD method produced estimated ITs and PIs with smaller biases and SDs. Consistent with the results of Simulation Study 1, the qCD method produced estimated ITs and PIs with higher accuracy and precision than the staircase method. The means and SDs of the estimated ITs and PIs of the three observers with five starting levels are summarized in Supplementary Table S4. 

Figure 11

 

Distributions of the estimated initial threshold (A) and percentage of improvements (B) in the transfer phase from the quick Change Detection and SC10% staircase methods. Blue asterisks and green circles denote results from the post hoc segment-by-segment quick Change Detection and staircase methods, respectively. Black dash-dotted lines denote the true values. Results from the three observers are shown in separate rows.

Estimated TI

The histogram of the estimated TIs of the three observers with the 0% starting level is shown in Figure 12. The histograms of the estimated TIs from the qCD and staircase method are both distributed symmetrically around the true value (= 50%), but the staircase method produced tighter distributions for Observers 1 and 2 (see Supplementary Figure S19 for starting levels of ±25% and ±50%). The estimated TIs from the qCD and staircase methods, averaged over starting levels, were respectively 39% ± 17% (M ± SD) and −121% ± 491% for Observer 1, 43% ± 14% and 25% ± 219% for Observer 2, and 47% ± 9% and 46% ± 48% for Observer 3. The results indicate that the qCD method provided more accurate and precise measures of transfer than the staircase method, especially when the learning is more rapid. 

Figure 12

 

Distributions of the transfer index of three simulated observers—(A) Observer 1, (B) Observer 2, (C) Observer 3—with a starting level at 0% from the true threshold in the first trial. Blue and green symbols denote the results from post hoc segment-by-segment quick Change Detection and SC10% staircase methods. Black dash-dotted lines denote the true values.

Discussion

In this study, we systematically examined the performance of the 3-down/1-up staircase and qCD methods in measuring the detailed time course of perceptual learning and transfer in a 2AFC task. The staircase method generated biased and imprecise estimations of the learning curve and transfer index, while the qCD method provided accurate and precise estimations. The staircase method did not converge when the block size was 40 trials. Larger block sizes (e.g., 160 trials) resulted in fewer block thresholds and therefore inaccurate and imprecise estimates of the parameters of the learning curves. Staircases with 1% and 5% step sizes failed to generate more than five reversals half of the time. Staircases with 10%, 20%, 30%, and 60% step sizes produced imprecise and biased estimation of thresholds and parameters of the learning curves. The staircase method could not provide an accurate and precise estimate of the transfer index even when the starting stimulus level was set optimally. The faster the learning was, the worse the staircase method performed. And the qCD method characterized the time course of perceptual learning and transfer more accurately, precisely, and efficiently than the staircase method. 

Zhang, Zhao, et al. (2018, 2019) implemented and tested the qCD method in assessing the learning curve in a 4AFC global motion-direction identification task both in simulations and in a psychophysical experiment. In their psychophysical experiment, the learning curves estimated from the qCD and staircase methods matched quite well for this relatively slow learning task. However, the qCD method still provided a more precise assessment of the learning curve. In their simulation experiment, with any one of three starting stimulus levels (+25%, 0%, and −25% from the true threshold), the estimated learning curves from both the trial-by-trial and post hoc segment-by-segment analyses of the qCD method were always more accurate and precise than those obtained from the staircase method using either 80- or 160-trial blocks. Going beyond previous work, the current simulation studies systematically evaluated the performance of the 3-down/1-up staircase method with six step sizes, five starting levels, and three block sizes on accessing the detailed time course and transfer of perceptual learning in a 2AFC task, which is the most widely used procedure in perceptual-learning studies (Jogan & Stocker, 2014; Kingdom & Prins, 2010; Vancleef et al., 2018). 

Step sizes of the staircase method were manipulated at six levels (1%, 5%, 10%, 20%, 30%, and 60%) to examine which step size could best track the detailed time course of perceptual learning. Smaller step sizes (1% and 5%) failed to generate reliable threshold estimates, because the staircase method does not generate enough reversals. Larger step sizes (20%, 30%, and 60%) produced block threshold estimates with lower accuracy and precision compared to the 10% step size. Although the commonly used 10% step size is the best option for the staircase method, it still produced estimates with substantial bias and SD for the estimated learning curve parameters in some cases. Furthermore, when learning is more rapid, the staircase method is less able to track the detailed time course of perceptual learning. This is not surprising, since there is an insufficient number of block measurements to track reductions in threshold, especially in the early rapid-changing phase of learning. 

In the current study, the starting levels (initial threshold levels tested) also influenced the performance of the staircase method in estimating learning curves. The biases and SDs of the estimated parameters from the staircase method depended more on the starting levels, while the qCD method was much less dependent on the deviation of the starting value from the true value. The staircase method performed worse when the starting levels were set nonoptimally. 

The staircase method generated biased and imprecise threshold estimates because a constant threshold is assumed within each block of trials, whereas the perceptual sensitivity during perceptual learning changes continuously (Lu et al., 2011; Mazur & Hastie, 1978; Petrov et al., 2005), even within each measurement block and especially in the early phase of learning (Badiru, 1992; Dosher & Lu, 2007; Heathcote et al., 2000). On the other hand, the qCD method parameterizes the learning curve as an exponential function with joint probability distributions of all three parameters. Based on the adaptive Bayesian framework (Kontsevich & Tyler, 1999; Lesmes, Jeon, Lu, & Dosher, 2006; Lesmes et al., 2010; Watson & Pelli, 1983), the qCD method selects the next test stimulus to optimize the expected information gain trial by trial. By continuously updating the posterior distribution trial by trial, the qCD method provides a detailed trial-by-trial estimate of perceptual sensitivity that tracks the detailed time course of perceptual-sensitivity change. Our simulations show that the estimated learning curves and their parameters from the qCD method were very accurate and precise with the 2AFC task tested here regardless of the deviations in starting levels. 

Although specificity is considered to be the hallmark of perceptual learning, a growing number of studies have found that learning can transfer and that the degree of transfer depends critically on the difficulty (Ahissar & Hochstein, 1997; Z. Liu, 1995; Z. Liu, 1999) or precision (Jeter et al., 2009) of the training or transfer tasks, training duration (Jeter, Dosher, Liu, & Lu, 2010), and training procedure (J. Huang, Liang, Zhou, & Liu, 2017; Liang et al., 2015; Tartaglia, Bamert, Mast, & Herzog, 2009; Xiao et al., 2008). Accurate and precise estimates of the initial and final thresholds in the learning phase and the initial threshold in the transfer phase are critical for computing the transfer index (Ahissar & Hochstein, 1997; Dosher & Lu, 2007; Jeter et al., 2010; J. Liu, Lu, & Dosher, 2012). In Simulation Study 1, the estimated initial threshold from the staircase method was quite biased and imprecise, while the qCD method yielded estimated initial thresholds with high accuracy and precision. In Simulation Study 2, we found that the estimated transfer index from the staircase method was much less accurate and precise than from the qCD method, especially when the learning was rapid. The qCD method performed well with all starting stimulus levels and learning rates. In these stimulations, we assumed that the learning rates were the same in the learning and transfer phases. Theoretically, transfer of perceptual learning could lead to a lower initial threshold or faster learning (Z. Liu & Weinshall, 2000). This could be even more challenging for the staircase method to estimate accurately. 

Although the priors in the current study were uniform rather than the weakly informative secant functions used by Zhang, Zhao, et al. (2018, 2019), the qCD procedure still produced estimates with relatively high accuracy and precision. Our qCD procedure is also superior to that of Kattner, Cochrane, and Green (2017). Although their procedure can provide an estimated continuous trial-by-trial learning curve that is more detailed than the block-by-block analysis typically used in perceptual-learning studies with the staircase method, the selection of test stimuli during learning was not optimized during data collection. As a result, the new fitting procedure significantly improved the confidence intervals of the estimated thresholds in the late phase of the learning curve, but did not substantially benefit estimates in the early phases of learning. 

Per the request of an anonymous reviewer, we rescored the simulated data from the staircase method using the Bayesian estimation component of the qCD method. The data of all three simulated observers (training part only) from the staircase method with three starting levels (0%, +50%, and −50% above the true initial threshold) and two step sizes (10% and 5%) were rescored using the trial-by-trial and post hoc segment-by-segment qCD procedures. For the data obtained with both step sizes, the average accuracy and precision (RMSE, HWCI, and SD) of the rescored thresholds were comparable to those of the direct estimates with the qCD method, except that the accuracy of the rescored thresholds of the simulated observer with the fastest learning rate, 5% step size, and +50% starting level were worse than those obtained with the qCD method directly (RMSE: 0.024 vs. 0.006 log₁₀ units; Supplementary Table S5; Supplementary Figures S20 through S23). These results are different from those of Zhao et al. (2019), who found that even after rescoring with the qCD method, the accuracy and precision of the estimated thresholds from the staircase methods were worse than those obtained directly with the qCD method. The difference between the two studies is that Zhao et al. investigated faster perceptual-sensitivity changes than we did (time constants of 20 vs. 40 trials). 

To further evaluate the effects of rescoring, we performed additional simulations on the training part of simulated Observer 1 (time constant = 40 trials) in 4AFC and 8AFC tasks and measured the performance with the staircase (step size = 10%) and qCD methods. In the 4AFC task, the starting levels were +22%, +84%, and −36% relative to the true initial threshold. In the 8AFC task, the starting levels were 0%, +50%, and −50% relative to the true initial threshold. The data obtained from the staircase method were rescored using the qCD method. We found that the average accuracy and precision of the rescored thresholds were comparable to those of the direct estimates from the qCD method, except when the starting level was much higher than the true initial threshold. For example, in the post hoc segment-by-segment analysis, the RMSEs of the estimated learning curve from the qCD and rescored staircase data were 0.002 and 0.005 log₁₀ units, respectively, in the 4AFC task and 0.001 and 0.008 log₁₀ units in the 8AFC task (Supplementary Table S6). 

The qCD method exhibited some advantages in estimating thresholds in the early phase of learning. Following the segment-by-segment analysis, in the 4AFC task the biases of the estimated threshold from the qCD method were −0.008, −0.009, and −0.009, respectively, after 1, 10, and 20 trials, and for the rescored staircase data they were −0.027, −0.024, and −0.021 (all in log₁₀ units); in the 8AFC task they were −0.003, −0.004, and −0.004 for the qCD method and −0.043, −0.038, and −0.032 for the rescored staircase data (Supplementary Figure S24). The same pattern of results was also true for the HWCIs of estimated thresholds. In the 4AFC task, the HWCIs of the estimated threshold from the qCD method were 0.050, 0.040, and 0.032 log₁₀ units, respectively, after 1, 10, and 20 trials, and for the rescored staircase data they were 0.063, 0.052, and 0.042 (Supplementary Figure S25). To examine the effect of the number of alternatives on the efficiency of the qCD method, we also evaluated the efficiency of the qCD method in 2AFC, 4AFC, and 8AFC tasks for Observer 1 (time constant = 40 trials). Defining efficiency as the number of trials for the procedure to reach criterion accuracy (RMSE) and precision (SD) levels (Zhao et al., 2019), we found that the qCD procedure required 93, 84, and 76 trials to reach 0.02 log₁₀-unit accuracy in 2AFC, 4AFC, and 8AFC tasks, respectively, and 677, 379, and 291 trials to reach 0.02 log₁₀-unit precision (see Supplementary Figures S26 and S27). Putting Zhao et al. (2019) and the current study together, we conclude the following: Rescoring the staircase-method data with the qCD method could lead to comparable accuracy and precision with direct qCD estimates when perceptual sensitivity changes relatively slowly, but not when it changes fast; the qCD method holds advantages in early phase of learning; and the qCD method is more efficient with more alternatives in forced-choice tasks. 

We focused on the staircase method in this study because it is the most widely used procedure in assessing effects of perceptual learning. Here we briefly discuss the difference between qCD and other adaptive procedures. QUEST is a classic adaptive Bayesian testing procedure for estimating threshold from a sequence of psychophysical trials (Watson & Pelli, 1979, 1983). It assumes a single stimulus dimension and two possible trial outcomes, and estimates a single psychometric-function parameter that is defined on the stimulus dimension. Recently, QUEST+ extended QUEST to allow estimation of multiple parameters of the psychometric function, more stimulus dimensions, and more trial outcomes, as well as any form of the psychometric function and flexible sampling of stimulus and parameter dimensions (Watson, 2017). However, the qCD method is quite different from QUEST. Like many existing adaptive psychophysical procedures, QUEST assumes that the threshold does not change over time or from trial to trial. The procedures are not designed to measure changing thresholds or perceptual sensitivity in situations such as dark adaptation and perceptual learning. Although it can be extended to assess the time-course perceptual-sensitivity change, the current implementation and applications of QUEST+ have focused on thresholds that do not change over time (Watson, 2017). Additional work is necessary to extend and evaluate QUEST+ in measuring the time course of perceptual-sensitivity change. 

Zhao et al. (2019) compared the performance of the qCD and quick Forced-Choice (qFC; Lesmes et al., 2015) methods in measuring the time course of dark adaptation. The qFC method, similar to QUEST and QUEST+, belongs to a family of novel Bayesian adaptive methods that are designed to estimate thresholds in yes/no and forced-choice tasks. Like QUEST and current implementations of QUEST+, the qFC method does not explicitly model perceptual-sensitivity change over time. Simulations showed that the accuracy and precision of the estimated dark-adaptation curve after one qCD run (RMSE = 0.002; HWCI = 0.016; SD = 0.020; all in log₁₀ units) were much higher than those obtained by 10 runs of the qFC procedure (RMSE = 0.020; HWCI = 0.032; SD = 0.031). 

In conclusion, we have systematically examined the performance of the 3-down/1-up staircase and qCD methods in measuring the detailed time course of perceptual learning and transfer in a 2AFC task. Our simulations demonstrated that the qCD method provides far more accurate and precise estimations of the learning curve and the transfer index than the staircase methods. 

Acknowledgments

This research was supported by the National Eye Institute Grants EY017491 and EY021553. 

Commercial relationships: Z-LL has an equity interest in Adaptive Sensory Technology, Inc., San Diego, CA, USA (I); YZ and Z-LL have an intellectual-property interest in methods for measuring behavioral changes of processes, PCT/US18/21944 (P). 

Corresponding author: Zhong-Lin Lu. 

Email: . 

Address: Laboratory of Brain Processes (LOBES), Department of Psychology, The Ohio State University, Columbus, OH, USA. 

Appendix A: The qCD procedure

Step 1. Threshold, psychometric function, and priors

In the qCD method, the true learning curve is characterized with a single exponential function:  

\begin{equation}\tag{A1}T\left( {\vec \theta ,n} \right) = \lambda exp\left( {{{ - n} \over \gamma }} \right) + \alpha {\rm ,}\end{equation}

where Display Formula\(\vec \theta = \left( {{\theta _1},{\theta _2},{\theta _3}} \right) = \left( {\lambda ,\gamma ,\alpha } \right)\), Display Formula\(n\) is the trial number during training, Display Formula\(T( {\vec \theta ,n})\) is the threshold at the Display Formula\(d^{\prime} = 1.5\) performance level on trial Display Formula\(n\), Display Formula\(\lambda \) is the dynamic range of learning, Display Formula\(\alpha \) is the asymptotic threshold level, and Display Formula\(\gamma \) is the time constant of the exponential function. Therefore, Display Formula\(\lambda + \alpha \) is the initial threshold before training. A broad joint prior distribution Display Formula\({p_0}( {\vec \theta } )\) is defined in the three-dimensional parameter space of Display Formula\(\vec \theta \), where the subscript 0 is used to denote that the prior that represents a priori knowledge of the parameters of the learning curve before the experiment.  

The prior distribution Display Formula\({p_0}( {\vec \theta } )\) was defined by the hyperbolic secant function  

\begin{equation}\tag{A2}{p_0}\left( {\vec \theta } \right) = {\rm{sech}}\left( {{\lambda _{confidence}}\left( {lo{g_{10}}\left( \lambda \right) - lo{g_{10}}\left( {{\lambda _{mode}}} \right)} \right)} \right) \times {\rm{sech}}\left( {{\gamma _{confidence}}\left( {lo{g_{10}}\left( \gamma \right) - lo{g_{10}}\left( {{\gamma _{mode}}} \right)} \right)} \right) \times {\rm{sech}}\left( {{\alpha _{confidence}}\left( {lo{g_{10}}\left( \alpha \right) - lo{g_{10}}\left( {{\alpha _{mode}}} \right)} \right)} \right)\end{equation}

where Display Formula\({\rm{sech}}\left( x \right) = 2/\left( {{e^x} + {e^{ - x}}} \right)\); Display Formula\({\lambda _{mode}}\), Display Formula\({\gamma _{mode}}\), and Display Formula\({\alpha _{mode}}\) are the peaks of the respective secant functions; and Display Formula\({\lambda _{confidence}}\), Display Formula\({\gamma _{confidence}}\), and Display Formula\({\alpha _{confidence}}\) are the spreads of the respective secant functions. The secant function is used as a weakly informative prior based on the previous literature (King-Smith & Rose, 1997) and results from a pilot study. The joint prior distribution Display Formula\({p_0}( {\vec \theta } )\) was updated trial by trial throughout the simulated experiment. A one-dimensional space Display Formula\(X\) covers all possible levels of the stimulus variation Display Formula\(x\left( { \in X} \right)\).  

In a two-alternative forced-choice task, the percentage-correct psychometric function is described by a Weibull function (Wichmann & Hill, 2001):  

\begin{equation}\tag{A3}p_n^{\prime} \left( {r = 1{\rm{|}}\vec \theta ,x} \right) = g + \left( {1 - g} \right)\left( {1 - {\rm{exp}}\left( { - {{\left( {{x \over {{T_w}\left( {\vec \theta ,n} \right)}}} \right)}^s}} \right)} \right){\rm ,}\end{equation}

 

\begin{equation}\tag{A4}{p_n}\left( {r = 1{\rm{|}}\vec \theta ,x} \right) = \left( {1 - \lambda } \right)p_n^{\prime} \left( {r = 1{\rm{|}}\vec \theta ,x} \right) + lg{\rm ,}\end{equation}

where Display Formula\(p_n^{\prime} \) is the psychometric function on trial n without a lapse rate, and Display Formula\({p_n}\) is with a lapse rate, Display Formula\(r\) is the response (1 for correct, 0 for incorrect), Display Formula\(x\) is the stimulus value, and Display Formula\(g = 0.5\) is the guessing rate. The slope of the Weibull function was set as Display Formula\(s = 3.06\) (Hou, Lesmes, Bex, Dorr, & Lu, 2015), and the lapse rate was set at Display Formula\(l = 0.04\). In this case, Display Formula\({T_w}\left( {\theta ,n} \right)\) is the Weibull threshold at trial Display Formula\(n\) given parameters Display Formula\(\vec \theta \):  

\begin{equation}\tag{A5}lo{g_{10}}\left({T_w}\left( {\vec \theta ,n} \right)\right) = lo{g_{10}}\left( {T\left( {\vec \theta ,n} \right)} \right) - {1 \over s}lo{g_{10}}\left( {\log \left( {{{1 - g} \over {1 - {p_{1.5}}}}} \right)} \right){\rm ,}\end{equation}

where Display Formula\({p_{1.5}} = 0.794\) is the probability correct corresponding to dDisplay Formula\(^{\prime} = 1.5\) in a two-alternative forced-choice task. Therefore, Display Formula\({p_n}( {r = 1{\rm{|}}\vec \theta ,x})\) is the probability of making a correct response Display Formula\(\left( {r = 1} \right)\) at trial Display Formula\(n\), conditioned on parameters Display Formula\(\vec \theta \) and the stimulus value Display Formula\(x\). The probability of making an incorrect response Display Formula\(\left( {r = 0} \right)\) is  

\begin{equation}\tag{A6}{p_n}\left( {r = 0{\rm{|}}\vec \theta ,x} \right) = 1 - {p_n}\left( {r = 1{\rm{|}}\vec \theta ,x} \right).\end{equation}

 

Together, Equations A1 through A6 describe the probability that any possible observer (defined by Display Formula\(\vec \theta \)) will make either a correct or an incorrect response in all possible stimulus conditions (Display Formula\(x\)) throughout the course of perceptual learning. The qCD method then uses an adaptive procedure to update the posterior distributions for a given observer based on their performance in the simulation. 

Step 2. One-step-ahead stimulus search

The stimulus in the next trial is selected from among all possible stimulus values (the one-dimensional space Display Formula\(X\)) to optimize the expected information gain (Kujala & Lukka, 2006; Lesmes, Jeon, Lu, & Dosher, 2006). The expected information gain of a potential test stimulus with Display Formula\(x\) is defined as  

\begin{equation}\tag{A7}{I_n}\left( x \right) = h\left( {\mathop \sum \limits_{\vec \theta } {p_n}\left( {\vec \theta } \right){p_n}\left( {r = 1{\rm{|}}\vec \theta ,x} \right)} \right) - \mathop \sum \limits_{\vec \theta } {p_n}\left( {\vec \theta } \right)h\left( {{p_n}\left( {r = 1{\rm{|}}\vec \theta ,x} \right)} \right),\end{equation}

 

\begin{equation}\tag{A8}h\left( p \right) = - p{\rm{log}}\left( p \right) - \left( {1 - p} \right){\rm{log}}\left( {1 - p} \right){\rm ,}\end{equation}

where h is the information entropy of the distribution p.  

Step 3. Bayesian update

After the stimulus for the next trial (e.g., trial Display Formula\(n\)) is selected by Step 2 and presented to the observer, the joint posterior distribution of the parameters of the exponential function is updated based on the response of the observer. 

In the nth trial of the learning curve, Display Formula\({r_n}\) is the observer's response to a stimulus with stimulus value Display Formula\({x_n}\) presented at trial Display Formula\(n\) of training, and Display Formula\({p_n}( {\vec \theta })\) is the prior distribution of parameters Display Formula\(\vec \theta \) at trial Display Formula\(n\) of training (equal to the posterior for trial n − 1). The prior distribution Display Formula\({p_n}( {\vec \theta } )\) is updated to the posterior distribution Display Formula\({p_n}( {\vec \theta {\rm{|}}{r_n},{x_n}} )\) via Bayes's rule:  

\begin{equation}\tag{A9}{p_n}\left( {\vec \theta {\rm{|}}{r_n},{x_n}} \right) = {{{\rm{\ }}{p_n}({r_n}|\vec \theta ,{x_n}){\rm{\ }}{p_n}\left( {\vec \theta } \right)} \over {{\rm{\ }}{p_n}({r_n}|{x_n})}}{\rm ,}\end{equation}

 

\begin{equation}\tag{A10}{p_n}({r_n}|{x_n}) = \mathop \sum \limits_{\mathop \to \limits_\theta } {p_n}({r_n}|\vec \theta ,{x_n}){\rm{\ }}{p_n}\left( {\vec \theta } \right){\rm ,}\end{equation}

where Display Formula\({{\rm{\Sigma }}_{\vec \theta }}\) is the summation over the three-dimensional parameter space—that is, Display Formula\({{\rm{\Sigma }}_{\vec \theta }} = {{\rm{\Sigma }}_\alpha }{{\rm{\Sigma }}_\lambda }{{\rm{\Sigma }}_\gamma }.\)  

The posterior distribution of Display Formula\(\vec \theta \) following the nth trial in the learning curve is used as the prior of trial n + 1:  

\begin{equation}\tag{A11}{p_{n + 1}}\left( {\vec \theta } \right) = {p_n}\left( {\vec \theta {\rm{|}}{r_n},{x_n}} \right){\rm .}\end{equation}

 

The marginal posterior distribution of each parameter is computed via summation of the joint posterior distribution over the other two parameters in the three-dimensional parameter space:  

\begin{equation}{p_n}\left( {\alpha {\rm{|}}{r_n},{x_n}} \right) = \mathop \sum \limits_\lambda \mathop \sum \limits_\gamma {p_n}\left( {\vec \theta {\rm{|}}{r_n},{x_n}} \right)\end{equation}

 

\begin{equation}\tag{A12}{p_n}\left( {\lambda {\rm{|}}{r_n},{x_n}} \right) = \mathop \sum \limits_\alpha \mathop \sum \limits_\gamma {p_n}\left( {\vec \theta {\rm{|}}{r_n},{x_n}} \right)\end{equation}

 

\begin{equation}{p_n}\left( {\gamma {\rm{|}}{r_n},{x_n}} \right) = \mathop \sum \limits_\alpha \mathop \sum \limits_\lambda {p_n}\left( {\vec \theta {\rm{|}}{r_n},{x_n}} \right){\rm {.}}\end{equation}

 

The expected values of the marginal posterior distributions provide estimates of the parameters of the exponential function after the nth trial in the learning curve:  

\begin{equation}\tag{A13}{\bar \theta _{f,n}} = \mathop \sum \limits_{{\theta _f}} {\theta _f} \cdot {p_n}\left( {{\theta _\alpha }{\rm{|}}{r_n},{x_n}} \right){\rm ,}\end{equation}

where Display Formula\({\theta _f} = \lambda ,\gamma ,\alpha \) for f = 1, 2, 3, respectively. (These are cast as summations rather than integrals because the parameter values for which these quantities are computed are quantized in the qCD method and in the simulations; see main text.)  

Based on the updated joint posterior distribution of parameters, the trial threshold can be estimated by the following procedures. We constructed 1,000 learning curves from the 1,000 sets of parameters randomly sampled based on the posterior probability distributions of the parameters. The average of the 1,000 learning curves was used as the threshold estimates. 

Step 4. Iteration of Steps 2 and 3

Steps 2 and 3 are iterated until a predefined number of training trials are completed. 

Step 5. Trial-by-trial and segment-by-segment analyses

Based on the joint posterior distribution after each trial, qCD can provide trial-by-trial estimates of the parameters and a single threshold in that trial. In addition, a post hoc segment-by-segment analysis can be used to partition the posterior distributions into segments based on their central tendency. The post hoc segment-by-segment analysis can determine whether the entire learning curve can be described by a single exponential function or a cascade of multiple exponent functions, and use all the information collected during each segment to further improve the accuracy and precision of the estimated thresholds (Zhao, Lesmes, & Lu, 2017, 2019). The posterior distribution obtained at the end of each segment is used to compute estimates of parameters and thresholds. 

We can partition the posterior distributions based on their central tendency across time. The distance between the central tendencies of the posterior distributions Display Formula\({p_{{n_1}}}({\vec \theta {\rm{|}}{r_{{n_1}}},{x_{{n_1}}}})\) and Display Formula\({p_{{n_2}}}({\vec \theta {\rm{|}}{r_{{n_2}}},{x_{{n_2}}}})\) at two trials, Display Formula\({n_1}\) and Display Formula\({n_2}\), is quantified by a modified Mahalanobis distance:  

\begin{equation}\tag{A14}MD\left( {{n_1},{n_2}} \right) = \sqrt {\vec y{C^{ - 1}}{{\vec y}^{tr}}} {\rm ,}\end{equation}

 

\begin{equation}\tag{A15}C = \left[ {{C_{{n_1}}} + {C_{{n_2}}}} \right]/2{\rm ,}\end{equation}

where Display Formula\(\vec y = \left[ {{y_1},{y_2},{y_3}} \right]\), Display Formula\({y_i} = {\bar \theta _{i,{n_1}}} - {\bar \theta _{i,{n_2}}}\), Display Formula\({\theta _f} = \lambda ,\gamma ,\alpha ,\)for f = 1, 2, 3, respectively; Display Formula\({\vec y^T}\) is the transpose of Display Formula\(\vec y\); Display Formula\({C^{ - 1}}\) is the inverse of the matrix Display Formula\(C\); and Display Formula\({C_{{n_b}}}\) (b = 1 or 2) is the Display Formula\(3 \times 3\) covariance matrix of the posterior Display Formula\({p_{{n_b}}}( {\vec \theta {\rm{|}}{r_{{n_b}}},{x_{{n_b}}}})\) for which the diagonal elements Display Formula\({c_{{n_b},{a_1}{a_1}}}\) and the off-diagonal elements Display Formula\({c_{{n_b},ij}}\) (iDisplay Formula\(,j \in \left[ {1,2,3} \right],i \ne j\)) are defined as  

\begin{equation}\tag{A16}{c_{{n_b},ii}} = Var\left( {{\theta _{i,{n_b}}}} \right) = \mathop \sum \limits_{\vec \theta } {\left( {{\theta _i} - {{\bar \theta }_{i,{n_b}}}} \right)^2} \cdot {p_{{n_b}}}\left( {\vec \theta {\rm{|}}{r_{{n_b}}},{x_{{n_b}}}} \right){\rm ,}\end{equation}

 

\begin{equation}\tag{A17}{c_{{n_b},ij}} = Cov\left( {{\theta _{i,{n_b}}},{\theta _{j,{n_b}}}} \right) = \mathop \sum \limits_{\vec \theta } \left( {{\theta _i} - {{\bar \theta }_{i,{n_b}}}} \right)\left( {{\theta _j} - {{\bar \theta }_{j,{n_b}}}} \right) \cdot {p_{{n_b}}}\left( {\vec \theta {\rm{|}}{r_{{n_b}}},{x_{{n_b}}}} \right){\rm .}\end{equation}

 

The null hypothesis is that the posterior distributions Display Formula\({p_{{n_1}}}( {\vec \theta {\rm{|}}{r_{{n_1}}},{x_{{n_1}}}})\) and Display Formula\({p_{{n_2}}}( {\vec \theta {\rm{|}}{r_{{n_2}}},{x_{{n_2}}}})\) are the same. We reject the null hypothesis when  

\begin{equation}\tag{A18}MD\left( {{n_1},{n_2}} \right) \gt M{D_0}{\rm ,}\end{equation}

where MD₀ is a predetermined criterion.  

To describe how we partition the posterior distributions of the dark-adaptation curve, we first introduce some notation: Display Formula\({u_l}\) is the trial number in the experiment, where l and Display Formula\({u_l}\) refer to the lth segment and the Display Formula\({u_l}\)th trial in the lth segment, respectively; Display Formula\({U_l}\) is the total number of trials in the lth segment; and L is the number of segments of the entire dark-adaptation curve. To partition the dark-adaptation curve into segments, we start from the last trial of the entire experiment, Display Formula\({U_L}\). The MD between Display Formula\({p_{{n_{{U_L}}}}}( {\vec \theta {\rm{|}}{r_{{U_L}}},{x_{{U_L}}}})\) and Display Formula\({p_{{n_{{u_L}}}}}( {\vec \theta {\rm{|}}{r_{{u_L}}},{x_{{u_L}}}} )\), the posterior distributions of the last trial and the previous trials, is calculated until Display Formula\({U_{L - 1}},\) the last trial of segment L − 1, is found:  

\begin{equation}\tag{A19}MD\left( {{p_{{n_{{U_L}}}}}\left( {\vec \theta {\rm{|}}{r_{{U_L}}},{x_{{U_L}}}} \right),{p_{{n_{{u_L}}}}}\left( {\vec \theta {\rm{|}}{r_{{u_L}}},{x_{{u_L}}}} \right)} \right) \le M{D_0},\forall {u_L} \in \left[ {1,{U_L} - 1} \right]{\rm ,}\end{equation}

 

\begin{equation}\tag{A20}MD\left( {{p_{{n_{{U_L}}}}}\left( {\vec \theta {\rm{|}}{r_{{U_L}}},{x_{{U_L}}}} \right),{p_{{n_{{U_{L - 1}}}}}}\left( {\vec \theta {\rm{|}}{r_{{U_{L - 1}}}},{x_{{U_{L - 1}}}}} \right)} \right) \gt M{D_0}{\rm ,}\end{equation}

where Display Formula\({n_{{u_L}}}\) and Display Formula\({n_{{U_L}}}\) refer to trials u_L and U_L (last trial) in the Lth segment (last segment).  

We then repeat the procedure to find all the segments. Therefore, for the Display Formula\(l\)th segment (Display Formula\(l\ \gt\ 1\)), we have  

\begin{equation}\tag{A21}MD\left( {{p_{{n_{{U_l}}}}}\left( {\vec \theta {\rm{|}}{r_{{U_l}}},{x_{{U_l}}}} \right),{p_{{n_{{u_l}}}}}\left( {\vec \theta {\rm{|}}{r_{{u_l}}},{x_{{u_l}}}} \right)} \right) \le M{D_0},\forall {u_l} \in \left[ {1,{U_l} - 1} \right]{\rm ,}\end{equation}

 

\begin{equation}\tag{A22}MD\left( {{p_{{n_{{U_l}}}}}\left( {\vec \theta {\rm{|}}{r_{{U_l}}},{x_{{U_l}}}} \right),{p_{{n_{{U_{l - 1}}}}}}\left( {\vec \theta {\rm{|}}{r_{{U_{l - 1}}}},{x_{{U_{l - 1}}}}} \right)} \right) \gt M{D_0}{\rm .}\end{equation}

 

Because the learning curve is continuous, we impose the following constraint on the posterior distributions between segments:  

\begin{equation}\tag{A23}{p_{{u_l}}}\left( {\lambda + \alpha } \right) = {p_{{n_{{U_{l - 1}}}}}}\left( {{f_{{t_{{U_{l - 1}}}}}}} \right){\rm ,}\end{equation}

where Display Formula\({p_{{u_l}}}\left( {\lambda + \alpha } \right)\) is the marginal prior distribution of Display Formula\(\lambda + \alpha \) at trial Display Formula\({u_l}\) (Display Formula\({u_l}\)th trial in the Display Formula\(l\)th segment) and Display Formula\({p_{{n_{{U_{l - 1}}}}}}( {{f_{{n_{{U_{l - 1}}}}}}})\) is the probability distribution of the threshold estimate at Display Formula\({n_{\left\{ {{U_{l - 1}},l - 1} \right\}}}\) of segment l − 1.  

Following segmentation, we use the posterior in the last trial of each segment, Display Formula\({p_{{n_{{U_l}}}}}( {\vec \theta {\rm{|}}{r_{{U_l}}},{x_{{U_l}}}} )\), to compute the estimated thresholds in the entire segment. Therefore, the posteriors are conditional on the data from the beginning of the experiment.