NotesFAQContact Us
Search Tips
ERIC Number: ED507804
Record Type: Non-Journal
Publication Date: 2009-Aug
Pages: 48
Abstractor: As Provided
Reference Count: 34
ISSN: ISSN-1930-9317
How Do Raters from India Perform in Scoring the TOEFL iBT[TM] Speaking Section and What Kind of Training Helps? TOEFL iBT[TM] Research Report. RR-09-31
Xi, Xiaoming; Mollaun, Pam
Educational Testing Service
This study investigated the scoring of the Test of English as a Foreign Language[TM] Internet-based Test (TOEFL iBT[TM]) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on-site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.-based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non-Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters' levels of confidence in scoring Indian examinees. Appendices include: (1) Distribution Statistics of the Scoring Samples in Scoring Session 1; (2) Distribution Statistics of the Scoring Samples in Scoring Session 2; and (3) Distributions of the Agreements Between Scores of ETS Raters and Raters From India in Scoring Session 1 With Regular Training. (Contains 13 tables and 4 notes.)
Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site:
Publication Type: Reports - Evaluative
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Educational Testing Service
Identifiers - Location: India; North America
Identifiers - Assessments and Surveys: Test of English as a Foreign Language