ERIC Number: EJ1168681
Record Type: Journal
Publication Date: 2017-Dec
Pages: 17
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-2330-8516
EISSN: N/A
Developing a Machine-Supported Coding System for Constructed-Response Items in PISA. Research Report. ETS RR-17-47
Yamamoto, Kentaro; He, Qiwei; Shin, Hyo Jeong; von Davier, Mattias
ETS Research Report Series, Dec 2017
Approximately a third of the Programme for International Student Assessment (PISA) items in the core domains (math, reading, and science) are constructed-response items and require human coding (scoring). This process is time-consuming, expensive, and prone to error as often (a) humans code inconsistently, and (b) coding reliability in international large-scale assessments varies across items and countries. The shift in PISA 2015 from paper- to computer-based assessment digitized all responses and associated coding, thus providing opportunities to introduce technology and analytical methods to improve data processing and analyses in future cycles. In this computer-based setting, this research explains the framework and approach for improving the accuracy and efficiency of the coding process in constructed-response items for future PISA cycles. In the pilot study associated with this research, we demonstrate how the proposed machine-supported coding system can be utilized using sample reading items from the PISA 2015 cycle.
Descriptors: Foreign Countries, Achievement Tests, International Assessment, Secondary School Students, Test Items, Responses, Scoring, Computer Assisted Testing, Test Scoring Machines, Reading Tests, Interrater Reliability, Accuracy, Efficiency
Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Publication Type: Journal Articles; Reports - Research
Education Level: Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: Australia; China; France; Germany; Japan; South Korea; Netherlands
Identifiers - Assessments and Surveys: Program for International Student Assessment
Grant or Contract Numbers: N/A