NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ797380
Record Type: Journal
Publication Date: 2005-Jul
Pages: 43
Abstractor: Author
Reference Count: 43
Evaluating Computer Automated Scoring: Issues, Methods, and an Empirical Illustration
Yang, Yongwei; Buckendahl, Chad W.; Juszkiewicz, Piotr J.; Bhola, Dennison S.
Journal of Applied Testing Technology, v7 n3 p1-43 Jul 2005
With the continual progress of computer technologies, computer automated scoring (CAS) has become a popular tool for evaluating writing assessments. Research of applications of these methodologies to new types of performance assessments is still emerging. While research has generally shown a high agreement of CAS system generated scores with those produced by human raters, concerns and questions have been raised about appropriate analyses and validity of decisions/interpretations based on those scores. In this paper we expand the emerging discussions on validation strategies on CAS by illustrating several analyses can be accomplished with available data. These analyses compare the degree to which two CAS systems accurately score data from a structured interview using the original scores provided by human raters as the criterion. Results suggest key differences across the two systems as well as differences in the statistical procedures used to evaluate them. The use of several statistical and qualitative analyses is recommended for evaluating contemporary CAS systems. (Contains 10 tables and 2 footnotes.)
Association of Test Publishers. 601 Pennsylvania Avenue NW, South Building, Suite 900, Washington, DC 20004. Tel: 866-240-7909; Fax: 717-755-8962; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Evaluative
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A