This paper investigates how an online test ("Self-Test" developed at the University of Southern Queensland) can enrich students' understanding of their academic numeracy, through a purpose-built, self-assessment tool aligned with online modules. Since its creation and evaluation, the tool has been developed and tailored to suit other first year courses based around an academic numeracy framework of competence, confidence and critical awareness (Galligan 2013a). This paper will highlight how the new Self-Test is underpinned by this framework and how students' levels of numeracy can be better understood by the lecturer through Self-Test in a first year nursing for numeracy course and a maths for teachers course. It particularly addresses over- and under-confidence, error analysis and students' reflective comments, and how this understanding can better inform course development and teaching.