Oral retellings are strongly recommended as a way to measure reading comprehension for second language learners (Bernhardt, 1985, 1990, 1991). However, the reliability of such ratings is a matter of concern for a variety of reasons (Aiken, 1996; Cooper, 1981; Saal, Downey, & Lahey, 1980). The purpose of this study was to establish reliable rating procedures to assess the reading comprehension of college-age second language learners using oral retellings. A three-facet (passage, rater, and rating occasion) generalizability study was conducted using oral retellings of expository passages. Retellings of three passages provided by 24 ELL students were rated by two trained raters on two occasions. The largest sources of error variance identified were the passages and the student-by-passage interaction. Results showed that the most important step that could be taken to reduce measurement error and increase generalizability would be to use at least four and preferably six passages to assess ELL students' reading comprehension. The gain in generalizability from using additional raters or rating occasions was minimal. (Contains 1 table and 3 figures.)