Ability grouping is a common practice in many mathematics classrooms. Its negative effects on students have been documented in research spanning two decades or more, and yet it is still widely implemented--in some countries even mandated. Using data from interviews with Australian students in the compulsory years of secondary schooling (14-16 year-olds), and using the theoretical tools offered by Bourdieu, I argue that the objective practices of school mathematics create an environment through which students internalize the practices to develop a sense of self, a "habitus". This "habitus" potentially is more or less empowering, depending on the experiences within these streamed settings.