Using 4 years of mathematics achievement scores, groups of typically achieving children (n = 101) and low achieving children with mild (LA-mild fact retrieval; n = 97) and severe (LA-severe fact retrieval; n = 18) fact retrieval deficits and mathematically learning disabled children (MLD; n = 15) were identified. Multilevel models contrasted developing retrieval competence from second to fourth grade with developing competence in executing arithmetic procedures, in fluency of processing quantities represented by Arabic numerals and sets of objects, and in representing quantity on a number line. The retrieval deficits of LA-severe fact retrieval children were at least as debilitating as those of the children with MLD and showed less across-grade improvement. The deficits were characterized by the retrieval of counting string associates while attempting to remember addition facts, suggesting poor inhibition of irrelevant information during the retrieval process. This suggests a very specific form of working memory deficit, one that is not captured by many typically used working memory tasks. Moreover, these deficits were not related to procedural competence or performance on the other mathematical tasks, nor were they related to verbal or nonverbal intelligence, reading ability, or speed of processing, nor would they be identifiable with standard untimed mathematics achievement tests. (Contains 6 tables.)