This study set out to establish the relationship between changes in episodic memory retrieval in normal aging on the one hand and gray matter volume and [superscript 18]FDG uptake on the other. Structural MRI, resting-state [superscript 18]FDG-PET, and an episodic memory task manipulating the depth of encoding and the retention interval were administered to 46 healthy subjects divided into three groups according to their age (young, middle-aged, and elderly adults). Memory decline was found not to be linear in the course of normal aging: Whatever the retention interval, the retrieval of shallowly encoded words was impaired in both the middle-aged and the elderly, whereas the retrieval of deeply encoded words only declined in the elderly. In middle-aged and elderly subjects, the reduced performance in the shallow encoding condition was mainly related to posterior mediotemporal volume and metabolism. By contrast, the impaired retrieval of deeply encoded words in the elderly group was mainly related to frontal and parietal regions, suggesting the adoption of inefficient strategic processes.