Retinal degenerative diseases like macular degeneration offer a means to investigate adult cortical plasticity in retinotopic visual cortex following topographically-constrained loss of retinal input. Functional neuroimaging studies of stimulus-driven activity in the cortical lesion projection zone (LPZ) have provided conflicting evidence regarding plasticity. Baker et al. (J. Neurosci. 2005) reported cortical remapping in a one-back task with object stimuli in two patients with impaired foveas, while Sunness et al. (Ophthalmology 2004) observed little or no remapping using a contrast-reversing checkerboard stimulus in a patient with a spared fovea. This suggests that the absence of foveal input is critical for remapping. Evidence from three patients reported by Masuda et al. (VSS 2007) provides an account of remapping in which postulated cortico-cortical signals modulate activity in the LPZ during a one-back task (but not during passive viewing), suggesting that LPZ activity depends on task rather than stimulus. We used fMRI in patients with no spared fovea during a blocked one-back task (using either full-field faces or scrambled faces in each block), and during viewing of full-field contrast-reversing checkerboards. On Baker's account, we should observe activity in the LPZ in all conditions (due to cortical remapping). On Masuda's account, we should observe remapping during the one-back task but not during checkerboard stimulation. We observed large inactive areas near the occipital pole for all patients during checkerboard stimulation. Furthermore, we observed activity in face-selective extrastriate cortex during face (but not scrambled-face) epochs of the one-back task, confirming face detection. Critically, a contrast of activity during the face versus scrambled-face epochs of the one-back task produced LPZ activity in some patients. Thus, our data suggest that when activity in the LPZ is observed, it is due to feedback from object-specific cortex, rather than cortical plasticity or the memory demands of the one-back task.