Background: Early intervention for children identified as late talkers (LTs) at the age of 24 months is still a controversial issue in research and clinical routine. Previous studies have shown inconsistent results regarding predictors of early lexical deficits on school-age outcomes of late-talking toddlers. Methods: In a five-wave follow-up study, we investigated various aspects of language and literacy abilities in 39 German-speaking third-graders who had been identified as LTs at the age of 24 months, compared to 39 typically developing children (TDC) also attending the third grade. The duration of auditory sensory memory was examined at the age of 4 years using mismatch negativity (MMN) of tones -- an event-related potential not confounded by any language skill. In addition, the predictive value of memory performance was examined in a longitudinal perspective. Results: Overall, LTs scored within normal range in language and literacy assessments. However, LTs differed from TDC in vocabulary size, verbalization of semantic relations, non-word repetition, and spelling. The findings can be explained by phonological working memory. The duration of auditory sensory memory and spatial working memory did not account for any variance. Conclusions: LTs sustain persistent phonological processing limitations even though their native language and literacy performance lay within the normal range at school age. Further research on second language acquisition, academic achievements, and the efficacy of early intervention in late-talking toddlers is recommended.