NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ780910
Record Type: Journal
Publication Date: 2007-Nov
Pages: 33
Abstractor: Author
Reference Count: 42
ISSN: ISSN-1530-5058
Evaluating Prototype Tasks and Alternative Rating Schemes for a New ESL Writing Test through G-Theory
Lee, Yong-Won; Kantor, Robert
International Journal of Testing, v7 n4 p353-385 Nov 2007
Possible integrated and independent tasks were pilot tested for the writing section of a new generation of the TOEFL[R] (Test of English as a Foreign Language[TM]). This study examines the impact of various rating designs and of the number of tasks and raters on the reliability of writing scores based on integrated and independent tasks from the perspective of generalizability theory (G-theory). Both univariate and multivariate G-theory analyses were conducted. It was found that (a) in terms of maximizing the score dependability, it would be more efficient to increase the number of tasks rather than the number of raters per essay; (b) two particular single-rating designs of "having different tasks for the same examinee rated by different raters" [p x (R:T), R:(p x T)] achieved relatively higher score dependability than other single-rating designs; and (c) a somewhat larger gain in composite score reliability was achieved when the number of listening-writing tasks was larger than that of reading-writing tasks. (Contains 4 figures, 6 tables and 8 footnotes.)
Lawrence Erlbaum. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Educational Testing Service, Princeton, NJ.
Authoring Institution: N/A