Prominent spoken language assessments such as the Oral Proficiency Interview and the Test of Spoken English have been primarily concerned with speaking ability as it relates to conversation. This paper looks at an additional aspect of spoken language ability, namely public speaking. This study used an adapted form of a public speaking rating scale originally designed for English as a foreign language (EFL) contexts. This paper seeks to evaluate the relationship between this EFL-based scale and assessment within a core US university public speaking course. The relevance of EFL assessments to those used in English medium courses provides insight as to whether language learners are being evaluated on similar public speaking constructs to their English speaking peers, and informs instruction aimed at preparing students for English medium courses. A sample of undergraduate students (N = 44), primarily native speakers of English, performed classroom speeches in an introductory public speaking course and were rated using the adapted public speaking rating scale. The rating scores, independent instructor evaluations, and written feedback from two raters were analyzed to determine reliability and validity, and to inform revisions of the scale. The rating scale was found to be reliable and valid for this population. Limitations of the scale and proposed expansions are discussed as the result of qualitative analysis of the data.