NotesFAQContact Us
Search Tips
ERIC Number: ED455253
Record Type: Non-Journal
Publication Date: 2001-Apr
Pages: 17
Abstractor: N/A
Reference Count: N/A
The Essay Scoring and Scorer Reliability in TOEFL CBT.
Lee, Yong-Won
An essay test is now an integral part of the computer based Test of English as a Foreign Language (TOEFL-CBT). This paper provides a brief overview of the current TOEFL-CBT essay test, describes the operational procedures for essay scoring, including the Online Scoring Network (OSN) of the Educational Testing Service (ETS), and discusses major psychometric issues related to the reliability of essay scores. Test takers can type or hand-write their written responses to the given essay prompt, and hand-written essays are scanned. All the writing samples are electronically distributed to trained ETS readers at regional sites through the OSN. The essays are scored online, and the scores are posted to the central database for use in score reporting. The two major sources of potential scoring error may come from the raters and the prompts. Readers undergo strict training, and prompts are reviewed thoroughly by content experts and monitored in terms of their performance through the OSN. It would be difficult to compute inter-reader reliability indices by using individual readers as units of reliability analysis, but rater reliability can be computed based on ratings, not readers, so that inter-reader reliability can possibly be examined at the levels of relative ratings, absolute ratings, and absolute readers. More research is necessary on the comparability of writing prompts. Future avenues for TOEFL-CBT essay scoring are automatic essay scoring and the development of new integrative writing tasks. (Contains 6 figures and 14 references.) (SLD)
Publication Type: Reports - Descriptive; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Test of English as a Foreign Language