A tetrachromatic display for the spatiotemporal control of rod and cone stimulation

Abstract

We present an apparatus that allows independent stimulation of rods and short (S)-, middle (M)-, and long (L)-wavelength-sensitive cones. Previously presented devices allow rod and cone stimulation independently, but only for a spatially invariant stimulus design (Pokorny, Smithson, & Quinlan, 2004; Sun, Pokorny, & Smith, 2001b). We overcame this limitation by using two spectrally filtered projectors with overlapping projections. This approach allows independent rod and cone stimulation in a dynamic two-dimensional scene with appropriate resolution in the spatial, temporal, and receptor domains. Modulation depths were ±15% for M-cones and L-cones, ±20% for rods, and ±50% for S-cones, all with respect to an equal-energy mesopic background at 3.4 cd/m². Validation was provided by radiometric measures and behavioral data from two trichromats, one protanope, one deuteranope, and one night-blind observer.

Introduction

Mesopic lighting conditions frequently occur in human environments, and often they are of artificial origin. They occur as a consequence of public lighting in urban areas, and they are sometimes intentionally created to prevent glare in otherwise dark environments. The latter includes street and traffic lighting but also indoor lighting in bars, cinemas, and theaters. Moreover, the visual displays of many mobile devices such as car dashboards, notebooks, and smart phones can be operated at upper mesopic intensity for convenient use under low lighting conditions. As a consequence, most visual content can be presented at mesopic levels. More than ever, this calls for a comprehensive understanding of mesopic vision. 

While the luminance of an artificial light source is almost linearly scalable, the sensory transduction from light to electrical impulses in the retina is less straightforward. At the retinal level, mesopic vision is characterized by the fact that both rods and cones contribute. More specifically, rod signals are fed into cone pathways within the retinal circuitry by at least two different mechanisms (see reviews by Lee, Martin, & Grünert, 2010; Sharpe & Stockman, 1999). This physiological entanglement of rods and cones has been observed to affect the perception of color (Buck, Knight, & Bechtold, 2000; Cao, Pokorny, & Smith, 2005; Cao, Pokorny, Smith, & Zele, 2008; Cao, Zele, & Pokorny, 2008), brightness (Stockman & Sharpe, 2006; Sun, Pokorny, & Smith, 2001a), motion (Gegenfurtner, Mayser, & Sharpe, 1999), and temporal processing (MacLeod, 1972; Sharpe, Stockman, & MacLeod, 1989; Zele, Cao, & Pokorny, 2008). 

Research on rod–cone interaction is complicated by the fact that common image devices provide only three primaries. Independent stimulation of rod and cone photoreceptors requires at least as many primaries as types of photoreceptors involved (Shapiro, Pokorny, & Smith, 1996)—that is, four primaries are necessary to study trichromats, whereas three primaries allow studying only participants who lack at least one type of cone receptor (e.g., Gegenfurtner et al., 1999; Kremers & Meierkord, 1999). However, it is not certain to what extent results from studies on cone-deficient individuals can be generalized to trichromats. 

A device with more than three primaries that has been used in research on rod–cone interaction in recent years is the two-channel four-primary photostimulator (Pokorny, Smithson, & Quinlan, 2004; Sun, Pokorny, & Smith, 2001b). The photostimulator presents a two-field stimulus consisting of a circular center and an annular surround by a Maxwellian view system. The spectral composition of both center and surround can be controlled independently by four light-emitting diodes (LEDs) each. This enables high contrasts of receptor-specific stimulation. Other devices with four or even more primaries have been used to investigate melanopsin-based receptors (e.g., Spitschan, Jain, Brainard, & Aguirre, 2014; Tsujimura, Ukai, Ohama, Nuruki, & Yunokuchi, 2010) or to generate illuminations with well-defined spectral characteristics (e.g., Pearce, Crichton, Mackiewicz, Finlayson, & Hurlbert, 2014). More recently, a photostimulator with five primaries was presented by Cao, Nicandro, and Barrionuevo (2015). With five primaries, rods, cones, and melanopsin-based photoreceptors can be differentially stimulated. 

Although these devices are well suited for investigating photoreceptor interactions in the temporal domain or the effects of large-scale illumination changes, their design does not allow dynamic variations of spatial parameters. However, spatiotemporal control is of key importance for vision research, and it appears to be highly crucial for the understanding of mesopic vision because of the known differences in both spatial and temporal tuning of rods and cones. It is highly desirable to investigate research questions addressing mechanisms of early visual processing as well as implications of rod–cone interaction when viewing natural scenes. 

To construct a display with custom primaries and spatiotemporal control, existing image devices have been modified and combined in the past—however, for objectives other than independent stimulation of rods and cones. Poirson and Maloney (1996) used three modified cathode ray tubes to yield a viewing system that can provide daylight intensities. Ajito, Obi, Yamaguchi, and Ohyama (2000) modified and combined two projectors to increase display gamut and improve color reproduction. The apparatus presented here follows the same approach by using two filtered LED projectors. These projectors, designed for industrial applications, offer detailed control over timing, intensity, and composition of the projected light. Through appropriate alignment and extensive spatial and spectral calibration, the combination of two of these projectors allows the stimulation of rods and cones independently with sufficient temporal and spatial resolution for mesopic vision research. 

Materials and method

In the following sections we describe the setup, alignment, spatial and spectral calibration, and computational procedures used to control the projectors in terms of photoreceptor excitation. 

Setup

Basic structure and components

The basic structure of the setup was a box-shaped, extruded aluminum frame. Two LED projectors (DLP LightCrafter 4500 Evaluation Module, Texas Instruments, Dallas, TX; 912 × 1140 pixels, 60 Hz) were mounted at the front side, one at the bottom and the other at the top (see Figure 1). Each projector was enclosed in a separate air-cooled casing to dissipate heat and absorb scattered light. The projection screen was embedded into the frame's back side. The resulting distance between the subject's eyes and the screen was 115 cm. A head and chin rest was used to control this distance. The frame's sides were lined with velour-covered walls to absorb scattered light from the projection screen. 

Figure 1

 

Basic structure (without walls and projector casings) during projector alignment showing frame, projectors, and screen as seen from the back side (left) and the front side (right). The robot was used to align the projectors.

Projectors

The projectors were based on digital light processing (DLP). The image-forming component of DLP is a digital micromirror device (DMD). A DMD contains an array of tiny mirrors, where one mirror corresponds to one pixel. These mirrors can rapidly toggle between two states, reflecting light from a global light source either through the lens (on state) or to an absorber (off state). Gray values are realized by binary pulse width modulation. The DLP4500 DMD we used provided 1140 × 912 square-shaped aluminum mirrors with a side length of 7.6 μm. The light sources were three LEDs peaking at 455 (blue), 520 (green) and 624 (red) nm (see Figure 2). To provide the perception of a multicolored image, the LEDs were turned on alternately in intervals of a few milliseconds. Hence, each video frame comprised subframes of one color, which in turn comprised pulses corresponding to the intensity (see Figure 3). 

Figure 2

 

Light engine of a single projector (top view). Light (illustrated by the yellow path) emerges from three LEDs (1–3), is reflected by or passes through dichroic mirrors (4), and is spectrally filtered (5). The filter is not a part of the original light engine but rather was added to modify the primaries' spectra. All primaries of a projector were filtered by the same filter; however, different filters were used for each projector. Mainly the green primary was differentially affected, whereas the blue primary in projector 1 and the red primary in projector 2 were less affected by the filters. The filters were placed directly in front of a mircolens array (6), which homogenizes the light and thus neutralizes the stray light caused by the filter. The parallelized light is mirrored (7), passes a total internal reflection (TIR) prism (8), and is reflected by the DMD (9), which actually forms the image. Light is reflected from pixels in the on state to the inner surface of the TIR prism, from where it leaves the light engine through the projection lens (10).

Figure 3

 

Left: Two DMD pixels. Mirrors can tilt perpendicular to the hinge axis (red) by ±12°. Depending on the direction, the pixel reflects light either on the screen or on an absorber. This graphic was taken from Douglass, M. R. (1998) Lifetime estimates and unique failure mechanisms of the digital micromirror device (DMD). 1998 IEEE International Reliability Physics Symposium Proceedings (pp. 9–16). Piscataway, NJ: Institute of Electrical and Electronics Engineers, courtesy of the author and with permission of the IEEE. Right: Exemplary light pulses corresponding to the intensities B1 = 108, G1 = 131, R2 = 119, and G2 = 108 (data based on photodiode measurements).

Common DLP projectors apply automatic video processing functions to the input video stream such as gamma, color transformations, and spatial filtering. Thus, an appropriate calibration is difficult to achieve with a common projector. The LightCrafter 4500 can be operated with disabled video processing functions, and various parameters can be controlled explicitly. The most important for our purpose were the partitioning of video frames into subframes, the assignment of LEDs to these subframes, and the LED currents. The number of subframes determines the temporal frequency by which LEDs alternate and the maximal bit depth per video frame that can be achieved; the higher the number of subframes, the lower the maximal bit depth. Here, a video frame was divided into two subframes, which provided eight bits per subframe. Each subframe was assigned to a single LED, which was blue (B1) and green (G1) in one projector and red (R2) and green (G2) in the other—that is, two primaries were used per projector. Because the projectors were synchronized by the video signal, primaries were shown in alternating pairs (B1–R2 and G1–G2; see Figure 3). 

This temporal configuration is preferred because it yields the highest bit depth per video frame. However, it is not suitable for tasks demanding saccades because a temporal breakup of primaries can be perceived during saccades by some observers. In such tasks a higher number of subframes should be considered. In fixation tasks, as used here, no artifacts were observed. 

Spatial calibration

Spatial alignment

The projected images must overlap to provide a display with more than three primaries. The more accurate this alignment, the higher is the available spatial resolution. Because the projectors' lenses have a vertical offset, their images can be combined by mounting one projector upside down above the other. This arrangement does not require tilting any projector relative to the screen, which would result in parts of the image being out of focus, and it does not require any additional optical element such as a beam splitter. However, the quality of alignment is limited because of the image distortion, which is nonsymmetric because of the lens offset. Because one image is upside down, the distortions of both images do not fit together; a pixel-accurate alignment can therefore not be achieved in principle. Thus, we aligned the projectors in good approximation and controlled the pixels based on their actual position on the screen. 

The alignment was done with a manually operated industrial robot (KR 120 R2500 Pro, KUKA AG, Augsburg, Germany; see Figure 1). The robot can be controlled with 6 df and has a pose repeatability of ±0.06 mm. During the alignment procedure, a dashed grid was projected by both projectors. The alignment was stopped when these grids were overlapping on the scale of single pixels, apart from the edges and corners, where the distortion was largest. Once in this position, the projectors were mounted to carrier plates on the aluminum frame and detached from the robot. Note that a robot is a convenient positioning tool but not a requirement. Other positioning methods can in principle yield qualitatively similar results. 

Measurement of pixel positions

An industrial camera (Basler acA2040-90um, Ahrensburg, Germany) was used to determine the pixel positions for both projectors. In short, a calibration table was photographed in the screen plane as a real-world reference. Pixel arrays were then projected and photographed for both projectors. Positions of reference markers and projector pixels were estimated in the same coordinate system (i.e., as image coordinates of the camera). Based on that, projector pixel positions could be estimated in real-world coordinates. 

The calibration table shows 71 × 51 white disks on a black background. The disks have a 5-mm diameter and are aligned in a rectangular grid with a 10-mm distance in the horizontal and vertical directions. The projected pixel arrays consisted of 143 × 113 single pixels with a grid spacing of six pixels in the horizontal direction and 10 pixels in the vertical direction. Due to the diamond pixel geometry of the DLP4500 DMD, odd and even rows are shifted horizontally with respect to each other (see Figure 4). Thus, two pixel arrays were photographed for each projector—one for odd rows and one for even rows. The centers of reference markers and projector pixels were located by (a) removing static noise by subtracting a dark image, (b) filtering with a difference of Gaussians, (c) estimating a threshold based on the brightness distribution, (d) computing the center of gravity for each blob of connected camera pixels above threshold, (e) estimating the grid vectors as the four most frequent vectors to neighboring blob centers, (f) assigning blob positions to grid positions by searching neighbors with the grid vectors, and (g) discarding outliers at the edges that did not form complete rows or columns. The image coordinates of projector pixels were mapped to real-world coordinates by linear interpolation of the reference markers. 

Figure 4

 

Left: The diamond pixel geometry with pixel indices (x, y). The horizontal pixel distance is twice the vertical distance, odd rows (red) are horizontally shifted with respect to even rows (green), and columns are jagged (blue). Right: Illustration of the image processing that yields the pixel positions (see description in text). It shows a section of two overlaid photographs with 2 × 2 calibration pixels each—one for odd rows (red) and one for even rows (green); yellow appears where both overlap. Black dots show the derived centers of projector pixels (large dots = centers of calibration pixels; small dots = centers of interpolated pixels). The blue diagonal lines show the reconstructed pixel boundaries.

The purpose of determining pixel positions is to control stimulation in real-world coordinates. Furthermore, the pixel positions can be used to estimate the spatial characteristics of the display. In the following we report the pixel size, the distance between overlapping pixels, and the spatial power spectrum within the 26° × 18° area that was used in psychophysical experiments; all angular values are relative to the subject's viewing distance of 115 cm. The pixel side length was 1.54 ± 0.03 arcmin (M ± SD). The distance between most overlapping pixels was 0.36 ± 0.14 projector pixel, which is equivalent to 0.55 ± 0.21 arcmin. The maximal observed distance was 0.71 projector pixel, which is one half a pixel diagonal; in this case, the pixels of both projectors were exactly interleaved. However, most interesting for vision research is the maximal spatial frequency that can be presented, or, more generally, the amplitude characteristic of the display. To estimate this function we simulated the downsampling of high-resolution white-noise images by the projectors' pixels. These images had a resolution that was 10 times higher than those of the projectors; by sampling them at lower resolution, high frequencies were dampened. The ratio of the power spectra of the downsampled images to the original images provides an estimate of the power characteristic of the display, for which ratios of 10,000 images were averaged. According to this simulation, the spatial frequency with half-power output, which corresponds to about 70% of the input amplitude, was 15 cycles/deg (see Figure 5). 

Figure 5

 

Left: Minimal distance between pixels of both projectors across the screen. The wave-like patterns are caused by the interference between the slightly different positions of projected pixels. Pixels perfectly overlap at the centers of black regions. The maximal pixel distance occurs where the bright lines intersect; here, pixels of both projectors are interleaved. Right: Spatial power characteristic of the combined projections. The half-power cutoff frequency is 15 cycles/deg (red line).

Spectral calibration

The spectral calibration was based on a linear model of four primaries (Shapiro et al., 1996), whose main ideas is reviewed briefly in the following section. Subsequently, we describe how we accounted for the limited bit depth per pixel, the spatial variation of power across the display, and stray light. All radiometric measurements mentioned here were made with a PR-650 Spectrascan Colorimeter (Photo Research, Chatsworth, CA), which was cross-validated with a Konica Minolta CS-2000 (Tokyo, Japan) beforehand. The corresponding photoreceptor absorption rates were computed using the 10° cone fundamentals by Stockman and Sharpe (2000) and the scotopic luminosity function (Wyszecki & Stiles, 1982), unless otherwise stated. 

The four-primary method

Our intention was to control the stimulation of photoreceptors, which was modeled as the absorption rate. The absorption rate b_i of photoreceptor i corresponding to a stimulus with spectral power distribution y(λ) is given as , where r_i(λ) describes the spectral sensitivity of the photoreceptor (the scaling effect of pupil size is not considered at this point). Here, we consider four photoreceptors. Thus, a stimulus can be written as a four-dimensional vector b. We refer to the corresponding vector space as the photoreceptor domain or the photoreceptor space.  

Consider a stimulation provided by a single primary. We assume that the primary's power distribution is scaled by its intensity. Scaling of spectral power yields equivalent scaling of the corresponding absorption rates. Hence, the stimulation can be written as a scaled vector in the photoreceptor domain. Here, this vector is referred to as the primary's absorption ratio a. It is defined as the stimulation provided by the primary at maximal intensity. The scaling factor corresponding to a certain intensity is referred to as the primary weight w. That is, the stimulation provided by a scalable single primary can be written as b = wa. 

Consider a stimulation provided by multiple primaries. Assuming that primaries are independent of each other, the resulting stimulation can be written as b = w₁a₁ + . . . + w_na_n, where n is the number of primaries. This is equivalent to the matrix notation b = Aw, where A is a 4 × n matrix whose columns are the primaries' absorption ratios and w is an n × 1 vector of primary weights. This describes a system of four linear equations, which is solvable for any target stimulation b if the matrix A contains at least four linearly independent columns (i.e., if at least four primaries with linearly independent absorption ratios are available). 

The same conclusion can be reached by regarding a stimulus as a point in the photoreceptor space. Independent stimulation of photoreceptors means that this point can be moved into any direction. To move freely in a four-dimensional space, at least four linearly independent vectors of movement are required. If, for instance, only three vectors are available, the accessible space will be limited to a three-dimensional hyperplane; movements perpendicular to this hyperplane are not possible (see Figure 6). The vectors of movement correspond to the absorption ratios of the primaries; thus, four primaries with linearly independent absorption ratios are required to independently stimulate rods and cones. Note that linearly independent spectra are not quite sufficient in principle; although they are necessary, they can in some cases result in linearly dependent absorption ratios, as apparent from metamers. 

Figure 6

 

Illustration of mapping target stimulation in the receptor domain (simplified to three dimensions). Dark arrows = receptor axes; bright arrows = absorption ratios of primaries; grid = subspace spanned by primaries; red transparent sphere = envelope of target subspace; red dots = mapped target subspace with three bits per primary. Left: The accessible space is limited to a hyperplane if the number of primaries is one less than the number of receptors. Middle: Misaligned and too powerful primaries; the target is partially outside of the accessible space, and the resolution is suboptimal. Right: Appropriately tuned primaries. We modified the primaries' direction by optical filters and changed their length by adjusting the LED current.

Tuning of primaries

The spectral power distributions of the unmodified green primaries G1 and G2 are close to identical and therefore linearly dependent. They have to be modified to enable independent stimulation of rods and cones. According to the four-primary method, any set of four linearly independent primaries is sufficient. However, we have to take into account that intensities are limited and discrete. As a consequence, it is not guaranteed that the target stimulation is within the operational range of the primaries or that it can be mapped with sufficient accuracy (see Figure 6). Consequently, the suitability of a set of primaries is determined by the target stimulation and the required level of accuracy. Here we selected the primaries—more precisely, their modifications—based on the simulated performance to provide a set of target stimulation. 

Two types of modification were applied: spectral filtering of the projectors and adjustment of LED currents. We simulated the effect of any pair-wise combination of 46 Lee HT polyester filters (Lee Filters, Burbank, CA) on the primaries at the screen center and optimized the currents for each set of filters individually. The effect of current on the power distribution is primarily scaling, which we modeled by third-degree polynomials from radiometric measurements beforehand. As a precaution, we limited the projectors' eight-bit current parameter to 40 for all LEDs, which corresponds to a current limitation of about 20% of the maximum. This excludes combinations of high LED current with low filter transmission, which could potentially damage the filters and LEDs due to overheating of the light engine. 

With the intention to measure temporal contrast sensitivities of rods and cones, the target stimulation was defined as positive and negative isolated photoreceptor contrasts with respect to an equal-energy background at 3.4 cd/m² and 150 to 230 scotopic trolands (for an estimated pupil diameter of 5.35 mm according to the pupil model of Watson & Yellott, 2012). This background stimulation is in the upper mesopic region, where rod-mediated contrast sensitivity is best according to Hess and Nordby (1986). A target was considered to be successfully represented if the error was less than 0.5% per photoreceptor. 

At first, the background stimulation was searched; in case a solution was found, the search was extended to increasing positive and decreasing negative contrasts in steps of 1%. The performance of a set of primaries was measured by the minimal feasible modulation depth. The best performance was achieved with Lee filters 161 and 161 for the projector with primaries B1 and G1 and Lee filters 121 and 21 for the projector with primaries R2 and G2 (see Figure 7). The LED current parameters were adjusted to 33, 38, 39, and 40, which corresponded to actual currents of 0.681 A, 1.065 A, 1.115 A, and 1.047 A (in the order B1, G1, R2, and G2). The resulting background provided a retinal illuminance of 176 scotopic trolands (for a pupil diameter of 5.35 mm). 

Figure 7

 

Top: Spectral power distribution of unfiltered primaries (solid lines; blue = B1, green = G1 and G2, red = R2) and nominal transmission spectra of filter (colored areas; blue = Lee 161 and 161, orange = Lee 121 and 21). Bottom: Spectral power distributions of filtered primaries (solid lines; blue = B1, cyan = G1, green = G2, red = R2) and receptor fundamentals (colored areas; blue = S-cones, cyan = rods, green = M-cones, red = L-cones) according to the 10° cone fundamentals reported by Stockman and Sharpe (2000) and the scotopic luminosity function (Wyszecki & Stiles, 1982).

Primary weights

The primary weights were derived from radiometric measurements of primaries and gray images by w = bA⁻¹, where b is the absorption rates corresponding to a gray image and A is the matrix of absorption ratios of the primaries. The relationship between intensity and primary weight had a linear characteristic except for a distinctive negative offset at intensity 128 (see Figure 8), which appears to be caused by a delayed LED onset. As a consequence, the first pulse in a subframe is shorter than intended. The affected pulse appears above intensity 127, where the offset is also observed. Although the size of the offset depends on the LED current, it is constant for a given current. Thus, it is sufficient for determining the primary weights for the currents used. In the special case of the selection of filters, where LED currents are free parameters, the primary weights were modeled by w = I/(2⁸ – 1), where I is the intensity. 

Figure 8

 

Primary weights. Left: Primary weights by radiometric measurements. The magnifications show the region around the mean intensity, where a vertical offset occurs. Right: Pulse sequences corresponding to intensities based on photodiode measurements. White shows on phases and black shows off phases of the pulse sequence. The onset of LEDs appears to be slightly delayed. Consequently, pulses that start with a subframe are too short. Such pulses are contained in pulse sequences of intensities above 127, which is in accordance with the position of the negative offset observed in the functions of primary weights.

Spatial variation of power

The spectral power of primaries dropped from a global maximum in direction of the corners by more than 30% (see Figure 9), which was caused by the directional characteristics of the LEDs. The spatial profile of power differed between primaries, where the most remarkable difference was caused by the fact that one projector operated upside down. As a consequence, brightness and color varied across the screen even for displays of intensities that were actually uniform. Measurements at the center and the edges of the screen revealed that primary spectra were scaled but not distorted. Hence, it is sufficient to determine a spatial weight v for each pixel and primary to account for the nonuniformity of the projection. To determine the spatial weights, we measured white circular fields with 1.2° diameter at 24 × 15 positions within an area of 805 × 1050 pixels corresponding to 29° × 19°. Spatial weights were estimated for each position of measurement by v = a a_ref⁻¹, where a is the absorption ratio at the regarded position and a_ref is the absorption ratio at the reference position (here the screen center). These weights were interpolated for each pixel within the measured area. The absorption ratio for any primary can be restored at any pixel within the calibrated area by a = v a_ref, where v is the spatial weight at the regarded pixel. 

Figure 9

 

Scaling of primaries' power across the screen. The power decreases from a maximum (cross) in direction to the corners. This is caused by the directional characteristics of the LEDs. The maximum is vertically displaced because of the vertical lens offset. The spatial distribution of power is flipped between projectors because one projector operates upside down (compare upper and lower rows).

Stray light

The effect of stray light is that of a spatial filter: Light at a certain location is mixed with light belonging to surrounding locations. Spatial dependency poses a serious challenge to calibration; it is advisable to avoid it as much as possible. Noncoated spectral filters as used here can cause substantial amounts of stray light. Thus, the filters were placed inside the light engine in front of the microlens array (see Figure 2). At this position, the stray light caused by the filters was homogenized by the mircolens array. In the psychophysical tests conducted in this study, one stimulus was shown at a time on a uniform background. Because the background was by far the largest stimulus to be presented, we assumed that stray light consisted primarily of misdirected background illumination. We measured the primaries within a small circular area (1.2° diameter) and a large rectangular area with the size of the background (26° × 18°). The increase of power with the large area compared with the small area is considered to be stray light, which was 1.3% for B1, 1.4% for G1, 1.1% for R2, and 1.1% for G2. Because the background was steady, stray light was regarded to be constant and was considered as baseline stimulation. 

Drift of LEDs

The spectral output of LEDs changed with temperature, which in turn depended on the time of operation. In repeated radiometric reference measurements during long-time calibration measurements, the radiance saturated at an increase of 2% to 3% after 2 hr. We corrected relative calibration measurements by such reference measurements. Psychophysical and absolute radiometric measurements were made after a warm-up phase of at least 120 min. 

Computation of intensities

According to the four-primary method, the optimal primary weights to target stimulation b were determined by w = (b – b_s)A⁻¹, where b_s is the baseline stimulation provided by stray light from the background and A is the matrix of absorption ratios of the pixel-specific primaries. The corresponding intensities were identified by the discrete weights with minimal distance to the optimal weights. The intensities corresponding to a stimulus were computed for each subframe, each pixel, and each projector separately. A subframe-wise approach provides the highest temporal resolution. A pixel-wise approach is necessary because of the nonuniform power of primaries across the screen. A projector-wise approach is necessary because the two sets of pixels, one from each projector, cannot be mapped to one set of pairs of pixels because some pixels share the most overlapping pixel. If pixels were grouped to pairs, multiple assignments would occur and ambiguous intensities would be possible. Thus, we created virtual pixels—that is, we modeled the primaries of one projector at all pixel positions of the other projector. The computation was done for pairs containing one real and one virtual pixel each. The outcome was four intensities for each pair. However, only the intensity of the primary of the real pixel, shown in the current subframe, was saved; the other three were discarded. 

The advantage of this method is that it treats pixels separately, which provides a simple approach to using redundancy and thus considerably reduces the computational effort. In a typical psychophysical experiment, a single pixel often takes the same or similar values, in particular for periodic and scaled stimuli as used here. To increase this redundancy further, the target absorption rates were rasterized in steps of 0.1% for rods, long (L)-wavelength-sensitive cones, and middle (M)-wavelength-sensitive cones and 0.25% for short (S)-wavelength-sensitive cones in terms of contrast to background. The computed intensities and the corresponding contrast values were saved in a database. By retrieving already-mapped targets from this database, the redundancy within (i.e., for different frames of a stimulus) and between similar stimuli was used to speed up the computation. 

Validation

The image device presented in this study was validated with radiometric and psychophysical measurements. The accuracy of the spectral calibration was tested with radiometric measurements. As a general test of this setup and its purpose in studying mesopic vision, temporal contrast sensitivities mediated by rod and cone photoreceptors were measured in a basic psychophysical experiment. 

Radiometric validation

We measured displays of uniform disks with 1.2° diameter at the screen center and at four screen locations at 5° eccentricity—the same at which stimuli were shown in the psychophysical experiment. The spectral power distributions of the disks corresponded to positive and negative contrasts of isolated receptor stimulation with respect to the mesopic background described earlier. The measured set of contrasts was ±15% in steps of 1.5% for L- and M-cones, ±20% in steps of 2% for rods, and ±50% in steps of 5% for S-cones. The disks were presented on top of the mesopic background to meet the identical stray light conditions that occurred during the psychophysical experiment. The absolute error of contrast was 0.22% ± 0.19% (M ± SD; see Figure 10). 

Figure 10

 

Radiometric validation. Each column shows measurements of a set of stimuli supposed to independently stimulate a single photoreceptor relative to a mesopic equal-energy background at 3.4 cd/m². The lower row shows the photoreceptor contrasts corresponding to the measured spectra (red = L-cone, green = M-cone, blue = S-cone, cyan = rod). In the ideal case, the contrasts of the isolated receptor would be perfectly aligned along the diagonal with positive slope and the contrasts of the other receptors would be zero. The upper row provides a detailed view of the difference from target contrast. Measurements were conducted at four locations used in the four-alternative forced-choice task and at the screen center.

Psychophysical validation

We measured the temporal contrast sensitivities of rods and cones with a four-alternative forced-choice task and the method of constant stimuli. The stimuli were two-dimensional Gaussian blobs (SD = 1°), which sinusoidally flickered in counterphase at either 1 or 10 Hz at one of four positions at 5° eccentricity. They were designed to stimulate exclusively L-cones, M-cones, S-cones, or rods. Two trichromats (one male, 33 years, and one female, 27 years; both authors of this study), two dichromats (one male protanope, 16 years, and one male deuteranope, 21 years), and one night-blind observer (male, 22 years) participated in the experiment. The trichromats were normal trichromats assessed with Ishihara pseudoisochromatic plates. The type of dichromacy was determined with the L- and M-cone flicker at 1 Hz and 15% contrast, which is suprathreshold for normal observers. The night-blind observer was assessed by an interview. He reported that his night vision is completely absent without any sign of adaptation, a condition that had been constant throughout his life. He is considerably myopic (−9 diopter, fully corrected with glasses) and had suffered from strabismus until a surgical correction in childhood. He reported that one of his brothers shows the same set of symptoms. This suggests that he suffers from X-linked congenital stationary night blindness type 1, where rods are dysfunctional. 

The experiment was divided into a demo session and a main session. The demo session took about 20 min and was conducted to familiarize the subjects with the task. Resulting data were not analyzed. The main session took approximately 70 min. The contrast range was adapted on the basis of earlier measurements except for the night-blind observer, who was tested with the full available contrast range for each photoreceptor. Prior to each session the subjects adapted to the mesopic light levels for 20 min. 

Eight conditions (four receptors at two temporal frequencies) were tested in separate blocks. The order of blocks was randomized for each subject. Each condition was tested with a scale of eight steps, where each step was measured equally often at each of the four possible positions—more specifically, once in the demo session and five times in the main session. The resulting number of repetitions per condition considered in analysis was 160. The order of stimulus position and contrast was randomized across trials. 

At the beginning of each trial the subject fixated a stationary Gaussian blob (SD = 0.075°, 30% contrast for L- and M-cones) presented at the screen center. After 700 ± 50 ms a stimulus was presented for 1000 ms at one of four possible positions at 5° eccentricity (see Figure 11). During the stimulus presentation the fixation blob remained visible; afterward, its polarity was inverted. This prompted the subject to specify the perceived stimulus position via a keypad. A trial was repeated at a later randomized time if the subject did not respond within 2 s. To maintain the subjects' attention, a short (45 s) break was conducted every 45 trials and between blocks during the main sessions. 

Figure 11

 

Left: Display in the four-alternative forced-choice flicker detection task. Gaussian blobs (SD = 1°), flickering with either 1 or 10 Hz, showing L-, M-, S-cone, or rod contrast could appear at one of four possible positions at 5° eccentricity on a 26° × 18° mesopic equal-energy background at 3.4 cd/m², which corresponds to 176 scotopic trolands (for an estimated pupil diameter of 5.35 mm). The fixation spot at the screen center was a constant 30% L+M Gaussian blob (SD = 0.075°). For better visibility the fixation spot is illustrated at double size. Right: Trial procedure. The trial started with the background and the fixation spot. After 700 ± 50 ms, a flicker appeared at one of the four positions for 1 s. After the presentation ended, the polarity of the fixation contrast was inverted, which prompted the subject to report the perceived position via a keypad.

Each block started with 16 (demo = 8) training trials to familiarize the subject with the type of stimulus of the following block. The training stimuli were presented with descending contrasts, whereby each step was shown twice in the main session and once in the demo session. After each training trial, subjects received auditory feedback about whether their response was correct or incorrect. No such feedback was given in regular trials, and training data were excluded from analysis. 

Results

The normal observers were most sensitive for the M- and L-cone flicker at 1 Hz (see Table 1 and Figure 12), whereas the sensitivity for S-cone-mediated stimulation was considerably less. All cone-mediated sensitivities declined from 1 Hz to 10 Hz. This drop of sensitivity was more pronounced for M- and S-cones than for L-cones. 

Table 1

 

Estimates of temporal contrast sensitivities for all photoreceptors and observers. Dashes indicate that the threshold performance (62.5% detection) was not reached at maximal contrast.

 

Figure 12

 

Temporal contrast sensitivities mediated by L-, M-, and S-cones and rods at 1 and 10 Hz for two trichromats (27 and 33 years), one protanope (16 years), one deuteranope (21 years), and one night-blind observer (22 years). The stimulation was presented on an equal-energy background at 3.4 cd/m², corresponding to 176 scotopic trolands for a pupil diameter of 5.35 mm. Error bars indicate the 95% confidence interval. Symbols are slightly shifted left or right to improve their visibility. Missing symbols indicate that the threshold performance (62.5% detection) was not reached at maximal contrast. The sensitivity of omitted data can be regarded as below 6.7 in the case of L- or M-cone stimulation, below 2 in the case of S-cone stimulation, and below 5 in the case of rod stimulation. Note the relative drop of sensitivity for cone flicker at 10 Hz for trichromats, the lack of sensitivity for the missing or dysfunctional receptor in the dichromats and the night-blind observer, the reversed sensitivity for L- and M-cones in the dichromats with respect to trichromats, and the lack of rod sensitivity at 1 Hz for all participants.

The dichromatic observers were not able to detect stimulation of the missing L- or M-cone, even at the highest contrasts reaching about 10 times the trichromats' threshold at 1 Hz. Both dichromats were more sensitive at 10-Hz than at 1-Hz stimulation of the remaining class of cone. However, the decline of S-cone-mediated sensitivity was similar in both trichromats and dichromats. 

The cone sensitivities of the night-blind observer were consistently lower than those of the trichromats, even though the principal pattern was similar. Rod stimulation could not be detected at either 1 Hz or 10 Hz. Although the other observers were able to detect rods at 10 Hz, with thresholds in the region of 5% to 15%, none of them were able to reach threshold performance for rod stimulation at 1 Hz (see Figure 13). 

Figure 13

 

Raw data and fitted cumulative Gaussians according to Figure 12. Note the baseline response of the protanope to L-cone stimulation, of the deuteranope to M-cone stimulation, of the night-blind observer to rod stimulation, and of all observers to rod stimulation at 1 Hz.

Discussion

Our goal was to construct an image device that allows independent spatiotemporal control of L-, M-, and S-cones as well as rods for the purpose of vision research. Ideally, such a device can be controlled pixel- and frame-wise with appropriate spatial and temporal resolution, similar to a normal display. According to Shapiro et al. (1996), this requires at least four primaries, which must have a linearly independent representation in the photoreceptor domain. 

Here we present a tetrachromatic image device with DLP projectors. DLP is based on binary pulse width modulation of light by an array of micrometer-scaled mirrors. It is one of the two major technologies used in commercial projectors. However, common projectors are not appropriate for vision research because they preprocess the video input with color transformations and spatiotemporal filters. This results in dependencies between the display primaries and spatial dependencies between pixels. The type of projector used here is designed for industrial and scientific applications as metrology, three-dimensional printing, and spectroscopy. Any preprocessing can be turned off, and basic parameters such as temporal frequency, bit depth, and LED current can be controlled directly. 

Two of these projectors were used, and two primaries were used from each projector. The vertical lens offset of the projectors was utilized to combine the projected images by operating one projector upside down above the other. We used an industrial robot to achieve an accurate alignment; however, the pixel arrays of both projectors could not be mapped perfectly because of nonmatching lens distortions. Nevertheless, a subpixel-accurate alignment was achieved by controlling the pixels based on their positions in real-world coordinates. These positions were determined with a calibrated industrial camera and an image processing algorithm developed for this task. The resulting display provided a field of view of 26° × 18° with a spatial resolution of 15 cycles/deg at half power. This is sufficient for a wide range of psychophysical examinations—in particular for the stimulation of nonfoveal regions where rod–cone interaction occurs. A higher spatial resolution can be realized by increasing the observer distance to the screen (or by decreasing the projector distance); however, this occurs at the expense of the field of view. 

To meet the requirement of four linearly independent primaries, each projector was equipped with a set of absorptive filters. The filters split the green primary into a short-wavelength part in one projector and a long-wavelength part in the other. They were selected based on the modeled performance of mapping a subset of the stimulation presented in the psychophysical tests. At the same time, the LED currents were adjusted to scale the primaries' dynamic range to the range of the target stimulation. 

Due to the directional characteristics of the LEDs, the power of the primaries declined toward the corners of the display. The spatial profile of power also differed between primaries. As a consequence, each pixel had a set of differently scaled primaries. We accounted for this by creating a spatial map of the primaries' power based on radiometric measurements across the screen. 

The video stream corresponding to the target stimulation was computed according to a linear model of four primaries (Shapiro et al., 1996) in a projector-, pixel-, and frame-wise approach. In this computation the effect of stray light of the psychophysical display, which was about 1% of the total radiance, was considered as constant baseline stimulation. This calibration approach was validated by measuring a set of isolated photoreceptor contrasts with a modulation depth of ±15% for M- and L-cones in steps of 1.5%, ±20% for rods in steps of 2%, and ±50% for S-cones in steps of 5% with respect to an equal-energy background at 3.4 cd/m². The measurements were conducted at the center and at the same four positions at 5° eccentricity as stimuli were presented during the experiments. The absolute error of contrast across receptors and positions was 0.22% ± 0.19% (M ± SD). 

However, the actual accuracy depends on the validity of the estimated spectral sensitivities of the photoreceptors. Systematic deviations in the spectral sensitivity functions cause systematic errors in calibration, and even small systematic errors can considerably bias thresholds—in particular if the target receptor is less sensitive than the receptors to be silenced. The determination of spectral sensitivity is complicated by its dependency on retinal location and variation between observers. The retinal location can be considered roughly by choosing either the 2° cone fundamentals for foveal stimulation or the 10° fundamentals for nonfoveal stimulation. This distinction is crucial because the macular pigment substantially alters the spectral sensitivity within the fovea. The stimuli in our experiment were shown outside the fovea; consequently, we used 10° cone fundamentals (Stockman & Sharpe, 2000). The variation between observers is caused mainly by different absorption characteristics of the cornea (van den Berg & Tan, 1994), lens (Cooper & Robson, 1969), and macula (Bone & Sparrock, 1971; Snodderly, Brown, Delori, & Auran, 1984). The absorption characteristic of the cornea and lens affects mainly the very short-wavelength edge of the visible spectrum. We simulated the effect of an intensity variation of the lens pigment by ±25% (a range of individual differences observed by Norren & Vos, 1974) based on the optical density function reported by Stockman, Sharpe, and Fach (1999). We found the resulting increase of calibration error to be negligible. The macular pigment can be neglected as well because it is absent outside the foveal region. 

To demonstrate the suitability of our apparatus for the investigation of mesopic vision, we measured temporal contrast sensitivities for rods and cones at 1 and 10 Hz in two trichromats, one protanope, one deuteranope, and one night-blind observer. The dichromats and the night-blind observer can be regarded as controls for the purity of the isolated stimulation of L-cones, M-cones, and rods. None of these subjects showed signs of detection for the missing or dysfunctional type of photoreceptor. 

The sensitivity for the remaining L- or M-cone in dichromats was higher for 10 Hz than for 1 Hz. This pattern of temporal sensitivity, which is reversed with respect to the trichromats, was previously reported for dichromatic observers by Sharpe, de Luca, Hansen, Jägle, and Gegenfurtner (2006). It is assumed to be a consequence of only one type of photoreceptor providing input into the magnocellular and parvocellular pathways, which causes an increased amount of either destructive or constructive interference. 

The temporal cone sensitivities of both trichromats are consistent with those measured with the four-primary photostimulator (Sun et al., 2001b) as well as with other measurements (L- and M-cones: Estévez & Spekreuse, 1974; S-cone: Stockman, MacLeod, & DePriest, 1991). The decline of cone-mediated sensitivities at 10 Hz in trichromats and the analogous decline for S-cone sensitivity in dichromats reflect the low-pass characteristic of the corresponding chromatic pathways (L- and M-cones: Kelly & van Norren, 1977; Smith, Pokorny, Davis, & Yeh, 1995; S-cone: Stockman et al., 1991). 

Rod stimulation at 1 Hz could not be perceived by any of the subjects. At 10 Hz, all but the night-blind observer detected rod stimulation with a threshold in the region of 5% to 15%. Conner (1982) and Sharpe et al. (1989) reported that the temporal contrast sensitivity of rods in trichromats changes from a low-pass profile at scotopic levels to a pronounced bandpass peaking in the region of 10 Hz at high mesopic levels. More specifically, rod sensitivity for low temporal frequencies steadily decreases with increasing luminance at upper mesopic levels, whereas sensitivity for frequencies about 10 Hz improves considerably. Sun et al. (2001b) measured rod sensitivity in trichromatic observers with the four-primary photostimulator. They found a similar, pronounced decline in rod sensitivity for low temporal frequencies when background luminance reached upper mesopic levels. This suggests that the maximal rod stimulation of 20% in our experiment still might be below threshold at 1 Hz. The lack of detection also indicates that the rod stimulation provided here did not contain a relevant proportion of cone stimulation because the trichromatic observers would have been highly sensitive to any L- and M-cone intrusion at 1 Hz. The same applies to rod stimulation at 10 Hz, where a subset of the stimuli shown at 1 Hz was used. 

The dissociation of rod sensitivity at 1 and 10 Hz with increasing background luminance could reflect different temporal characteristics of the two rod pathways (Sharpe & Stockman, 1999). The pathway mediated by rod bipolar and AII amacrine cells could account for the low-pass characteristics at scotopic up to moderate mesopic levels. At upper mesopic levels, this pathway might reach its operational limit, resulting in a progressing decline of rod sensitivity at low temporal frequencies. In contrast, the pathway mediated by rod–cone gap junctions could account for the sensitization for higher temporal frequencies starting at upper mesopic levels. Interestingly, the achromat lacks any dissociation of sensitivity for low and high temporal frequencies at upper mesopic levels. Rather, the scotopic low-pass profile of sensitivity continuously improves with luminance regardless of the temporal frequency. At the same time, data from critical flicker fusion experiments suggest that two different rod mechanisms exist in the achromat as well (Hess & Nordby, 1986). Thus, a simple two-pathway model might not be sufficient to explain all of the observed phenomena. 

Our results demonstrate that the isolation of photoreceptors was successful. The range of contrasts that we can achieve on our display is similar to that of the photostimulator (Pokorny et al., 2004) and is sufficiently high to deal with threshold stimulation at the mesopic level. Based on Conner (1982) and Sun et al. (2001b), we assume that the rod mechanism processing low temporal frequencies operates at its limits under the conditions tested here. We expect that at lower mesopic levels the contrast range of the device is sufficient for investigating rods at low temporal frequencies as well. The contrast and luminance range of the apparatus presented here can be widely modified by the selection of filters and LED currents as described in the Materials and method section. However, although our device is well suited for investigating rods, cones, and their interaction, we cannot achieve the high luminance levels that would be required to activate melanopsin receptors. 

With an increasing number of mesopic conditions in our environment, a better understanding of the mechanisms and characteristics of mesopic vision is of growing interest. The current method meets the requirements for studying the properties, function, and interaction of photoreceptors underlying mesopic vision in normal trichromatic subjects. In comparison with existing methods, which provide temporal control of receptor stimulation, the current device extends this control to the spatial domain. This offers the opportunity to investigate rod–cone interaction in motion and shape and to identify advantages and disadvantages of rod intrusion in complex visual tasks. The demands on construction are comparable to those of a typical psychophysical setup. The software demands for calibration and operation might exceed common procedures. However, the overall technical challenge is considered to be moderate. In this regard, a device like the one presented here is a powerful and feasible tool for exploring the role of rods in vision. 

Acknowledgments

This work was largely supported by Deutsche Forschungsgemeinschaft Grant GE 879/10 and Carl Zeiss Automated Inspection. 

Commercial relationships: none. 

Corresponding author: Florian S. Bayer. 

Email: [e-mail]. 

Address: Abteilung Allgemeine Psychologie, Universität Gießen, Gießen, Germany.