In the course of elementary school, children start to develop an academic self-concept reflecting their motivation, thoughts, and feelings about a specific domain. For the domain of mathematics, gender differences can emerge which are characterized by a less pronounced math self-concept for girls. However, studies are rather sparse regarding the early years of elementary school education, hence, the point in time when such gender differences emerge yet remains a matter of debate. In our study, we found that the math self-concept of elementary school children (n = 81) declined from first to second grade. While no differences in math achievement were observed between girls and boys, it became apparent that girls' math self-concept was already less pronounced than the math self-concept of boys in the first years of elementary school. Our findings emphasize the importance of considering such gender differences even at the beginning of school education.