Once a memory has formed, it is thought to undergo a gradual transition within the brain from short- to long-term storage. This putative process, however, also poses a unique problem to the memory system in that the same learned items must also be retrieved across broadly varying time scales. Here, we find that neurons in the ventrolateral prefrontal cortex (VLPFC) of monkeys, an area interconnected with both temporal and frontal associative neocortical regions, signaled the need to alter between retrieval of memories formed at different times. These signals were most closely related to the time interval between initial learning and later retrieval, and did not correlate with task switch demands, novelty, or behavioral response. Consistent with these physiological findings, focal inactivation of the VLPFC led to a marked degradation in retrieval performance. These findings suggest that the VLPFC plays a necessary regulatory role in retrieving memories over different temporal scales.