In this response we address some of the significant issues that Tony Brown raised in his analysis and critique of the Special Issue of "Educational Studies in Mathematics" on "Semiotic perspectives in mathematics education" (Saenz-Ludlow & Presmeg, Educational Studies in Mathematics 61(1-2), 2006). Among these issues are conceptualizations of subjectivity and the notion that particular readings of Peircean and Vygotskian semiotics may limit the ways that authors define key actors or elements in mathematics education, namely students, teachers and the nature of mathematics. To deepen the conversation, we comment on Brown's approach and explore the theoretical apparatus of Jacques Lacan that informs Brown's discourse. We show some of the intrinsic limitations of the Lacanian idea of subjectivity that permeates Brown's insightful analysis and conclude with a suggestion about some possible lines of research in mathematics education.