This study describes how oral language was assessed in an advanced-level college foreign language (FL) conversation course. Learners used the American Council on the Teaching of Foreign Languages Proficiency Guidelines to guide self-analyses of their oral production at intervals throughout the course. The intent was to provide opportunities for learners to develop an understanding of what constitutes oral proficiency, gauge their own progress, and set personal goals. Learners' self-analysis narratives suggested they began to notice different aspects of their speech and to better articulate their abilities and limitations. Broadly speaking, the results suggest that self-assessment of oral performance guided by the Proficiency Guidelines is an effective way to increase FL students' language awareness and self-efficacy. Pedagogical implications and limitations to this approach are discussed.