Abstract
To facilitate reasoning about 3D visual scenes our brain must convert complex visual inputs to simpler representations. While this conversion has been studied with psychophysics in humans, the underlying neural representations are poorly understood. Here we study how neural representations change as visual scenes are transformed by 3D rotations. Like images, representations at the level of the retina are “compositional”: The change of the representation to one transformation plus the change to another transformation is the same as the change to the combined transformation. We hypothesize that representations in the visual cortex are also compositional and texture independent. Based on this hypothesis we construct a geometric model in which we input neural activities to 3D-rotated planar surfaces and then predict the change in neural activities to multiples of those rotations. To test our hypothesis, we imaged the activity of more than 60,000 neurons in the visual cortex while a mouse is passively viewing planar textures at different 3D orientations projected to a monitor. Even though our textures differ widely from each other (e.g. in spatial frequency, “natural-ness” etc.), we find that 3D orientation is responsible for a large fraction of the variance in neural activity. In a dimensionally reduced subspace of neural activity that contains most of the variance, 3D orientation can be decoded linearly within the range of values that we tested. Finally, the geometric model fits our data well. These findings show that the mouse visual cortex encodes the 3D orientations of planar surfaces in a compositional and texture independent manner. We conclude that mice are a promising animal model for studying 3D vision, and that our theory provides a step towards understanding how visual scenes are encoded by the brain.