Mathematical competencies are important not only for academic achievement at school but also for professional success later in life. Although we know a lot about the impact of "Home Literacy Environment" on the development of early linguistic competencies, research on "Home Numeracy Environment" (HNE) and the assessment of its influence on the development of mathematical abilities is in its infancy. We still lack studies analysing this relationship and simultaneously controlling for other variables concerning the individual and the environment. Thus, in this article, we focussed on the development of mathematical competencies in a sample of 609 German children from the end of kindergarten until the end of Grade 1. In particular, we were interested in the role HNE plays in regard to this development while controlling for age, sex, intelligence, rapid naming, number span, linguistic competencies, kindergarten attendance and socioeconomic status. Moreover, HNE was compared between families with or without a history of mathematical disability. HNE was not only an important predictor of mathematical abilities at the end of kindergarten, but it also influenced the further development of mathematical competencies above and beyond its initial impact. Families with a history of dyscalculia provided a more unfavourable HNE than families with no such problems. Results are shown in a structural equation model, which highlights the importance of HNE. The findings indicate that those involved in policy and intervention should focus more on the learning environments in families to improve children's achievement.