Purpose: There is an ongoing need to develop assessments of spontaneous speech that focus on whether the child's utterances are comprehensible to listeners. This study sought to identify the attributes of a stable ratings-based measure of speech comprehensibility, which enabled examining the criterion-related validity of an orthography-based measure of the comprehensibility of conversational speech in students with Down syndrome. Method: Participants were 10 elementary school students with Down syndrome and 4 unfamiliar adult raters. Averaged across-observer Likert ratings of speech comprehensibility were called a "ratings-based" measure of speech comprehensibility. The proportion of utterance attempts fully glossed constituted an "orthography-based" measure of speech comprehensibility. Results: Averaging across 4 raters on four 5-min segments produced a reliable (G = 0.83) ratings-based measure of speech comprehensibility. The ratings-based measure was strongly (r > 080) correlated with the orthography-based measure for both the same and different conversational samples. Conclusion: Reliable and valid measures of speech comprehensibility are achievable with the resources available to many researchers and some clinicians.