ERIC Number: ED464963
Record Type: Non-Journal
Publication Date: 2002-Apr-4
Reference Count: N/A
Comparability of TOEFL CBT Essay Prompts for Different Native Language Groups.
Lee, Yong-Won; Breland, Hunter; Muraki, Eiji
Since the writing section of the Test of English as a Foreign Language (TOEFL) computer based test (CBT) is a single-prompt essay test, it is very important to ensure that each prompt is as fair as possible to any subgroups of examinees, such as those with different native language backgrounds. A particular topic of interest in this study is the question of whether the examinees of non-Indo-European native language backgrounds can be unduly disadvantaged on some essay prompts compared to those of European language backgrounds. A total of 81 TOEFL CBT writing prompts (262,034 essays) introduced from July 1998 through July 2000 were examined for essay prompt differential item functioning (DIF). The English language ability (ELA) variable was created by summing the standardized TOEFL Reading, Listening, and Structure scores and used to match examinees in the East Asian (Chinese, Japanese, and Korean) and European (German, French, and Spanish) native language groups. A three-step logistic regression modeling procedure for polytomous items was used to examine both the uniform and nonuniform effects of the native languages on the essay scores. Mean ELA, raw essay, and residual-based effect sizes were also computed for each prompt. Although approximately one-third of the prompts analyzed had statistically significant group differences (p=0.05), the R squared effect size was too small for any prompt to be classified as having a group effect. An appendix describes the derivation of the logistic regression model used. (Contains 5 tables, 4 figures, and 29 references.) (Author/SLD)
Publication Type: Numerical/Quantitative Data; Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: Educational Testing Service, Princeton, NJ.
Identifiers - Assessments and Surveys: Test of English as a Foreign Language