Face perception is thought to differ from object recognition by its reliance on configural, rather than part-based, processing. Here we examined the relative contributions of configuration and parts to early “face-selective” processing at the M170, a magnetoencephalographic (MEG) response occurring approximately 170 ms after stimulus onset, using adaptation. Previously (VSS 2005), we demonstrated that successive presentation of two stimuli with a brief (<800 ms) stimulus onset asynchrony results in attenuation of the M170 response. This adaptation is “face-selective,” with greater attenuation for faces preceded by faces than for faces preceded by houses. “Face-selective” M170 adaptation is not weakened even when the adapting face image is embedded in heavy visual noise (S/N < 20%). Our method therefore provides a new means of assessing the nature of the processing indexed by this early neurophysiological response. We measured the adapting power of face configurations versus face parts using upright and inverted faces, face-like configurations of black ovals versus scrambled non-face configurations of face parts, and isolated face parts. While face configurations alone do not produce “face-selective” adaptation, inverted faces, scrambled faces, and even isolated face parts adapt the M170 response to a similar extent as full faces. Thus, at least at the earliest stages of “face-selective” processing indexed by M170 adaptation, face parts undergo “face-selective” processing but face configurations do not. These results suggest that a part-based system precedes the putative configural stage of face processing.