NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1168681
Record Type: Journal
Publication Date: 2017-Dec
Pages: 17
Abstractor: As Provided
ISSN: EISSN-2330-8516
Developing a Machine-Supported Coding System for Constructed-Response Items in PISA. Research Report. ETS RR-17-47
Yamamoto, Kentaro; He, Qiwei; Shin, Hyo Jeong; von Davier, Mattias
ETS Research Report Series, Dec 2017
Approximately a third of the Programme for International Student Assessment (PISA) items in the core domains (math, reading, and science) are constructed-response items and require human coding (scoring). This process is time-consuming, expensive, and prone to error as often (a) humans code inconsistently, and (b) coding reliability in international large-scale assessments varies across items and countries. The shift in PISA 2015 from paper- to computer-based assessment digitized all responses and associated coding, thus providing opportunities to introduce technology and analytical methods to improve data processing and analyses in future cycles. In this computer-based setting, this research explains the framework and approach for improving the accuracy and efficiency of the coding process in constructed-response items for future PISA cycles. In the pilot study associated with this research, we demonstrate how the proposed machine-supported coding system can be utilized using sample reading items from the PISA 2015 cycle.
Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: Australia; China; France; Germany; Japan; South Korea; Netherlands
Identifiers - Assessments and Surveys: Program for International Student Assessment
Grant or Contract Numbers: N/A