This paper investigates the construction of systems of competence in two middle school mathematics classrooms. Drawing on analyses of discourse from videotaped classroom sessions, this paper documents the ways that "agency" and "accountability" were distributed in the classrooms through interactions between the teachers and students as they worked on mathematical content. In doing so, we problematize the assumption that competencies are simply attributes of individuals that can be externally defined. Instead, we propose a concept of individual competence as an attribute of a person's participation in an activity system such as a classroom. In this perspective, what counts as "competent" gets constructed in particular classrooms, and can therefore look very different from setting to setting. The implications of the ways that competence can be defined are discussed in terms of future research and equitable learning outcomes.