This paper makes both a critical analysis of some popular cultural texts about mathematics and mathematicians, and explores the ways in which young people deploy the discourses produced in these texts. We argue that there are particular (and sometimes contradictory) meanings and discourses about mathematics that circulate in popular culture, that young people use these as resources in identity making as (non-)mathematicians, negotiating their meaning in ways that are not always predictable, and that their influence on young people is diffuse and nevertheless important. The paper discusses the discourses that prevail in some of the popular cultural images of mathematics and mathematicians that came up in our research. We show how mathematics is represented as a secret language, while mathematicians are often mad, mostly male and almost invariably white. We then explore how young people negotiate these discourses, positioning themselves in relation to mathematics. Here we draw attention to the fact that both those who continue with mathematics after it ceases to be compulsory and those who do not, deploy similar images of mathematics and mathematicians. What is different is how they respond to and negotiate these images. (Contains 7 notes.)