Stereoscopic vision depends on the correct matching of corresponding features between inputs from two eyes, as false matching destroys the perception of depth. At what level the brain can discriminate correct matches from false ones is an important issue in stereoscopic vision. It has previously been shown that the majority of neurons in V1 tuned to correlated random dots stereograms(RDS) also responded to anticorrelated random dots stereograms (ARDS), a stimulus in which inputs to left and right eyes do not match with each other. Therefore, completion of the stereo matching problem must be solved at levels beyond V1. To examine this, we compared the neural population responses in three cortical areas, V1, V2, and V4 to RDS and ARDS in macaque monkeys. With the intrinsic optical imaging method, we found that ensemble activity patterns in V2 and V4 contain robust information related with stereoscopic depth. We are able to decode which of seven possible levels of horizontal disparities defined by RDS a monkey was viewing by a pattern classification method. Such information was not encoded in V1 as the correct rates of predictions based on activity patterns in V1 were close to random level. Furthermore, we tested whether disparity information defined by false matches were encoded in a similar way in V2 and V4. Seven disparity levels defined by ARDS were used. We found it is impossible to decide which horizontal disparities the monkey was viewing either from ensemble activity of V2 or V4. Thus, neurons in V2 and V4 respond to correct matches and reject false ones at least in population level. These results not only support and extend our previous findings regarding topographic representation of horizontal disparity in V2, but also indicate that the neural substrate for discarding stereoscopic false matches lies in early stages of extrastriate visual cortex.