The medial temporal lobe memory system matures relatively early and supports rudimentary declarative memory in young infants. There is considerable development, however, in the memory processes that underlie declarative memory performance during infancy. Here we consider age-related changes in encoding, retention, and retrieval in the context of current knowledge about the brain systems that may underlie these memory processes. While changes in infants' encoding may be attributed to rapid myelination during the first year of life, improvements in long-term retention and flexible retrieval are likely due to the prolonged development of the dentate gyrus. Future studies combining measures of brain and behavior are critical in improving our understanding of how brain development drives memory development during infancy and early childhood.