ERIC Number: ED442848
Record Type: Non-Journal
Publication Date: 1998-Apr
Reference Count: N/A
Classification Trees for Quality Control Processes in Automated Constructed Response Scoring.
Williamson, David M.; Hone, Anne S.; Miller, Susan; Bejar, Isaac I.
As the automated scoring of constructed responses reaches operational status, the issue of monitoring the scoring process becomes a primary concern, particularly when the goal is to have automated scoring operate completely unassisted by humans. Using a vignette from the Architectural Registration Examination and data for 326 cases with both human and computer scores available, this study reports on the usefulness of an approach based on classification trees (L. Breiman, J. Friedman, R. Olshen, and C. Stone, 1984) as a means of quality control. Five studies were carried out analyzing different aspects of the "training set" and making efforts to cross-validate the results of the analysis by applying the resulting classification trees to data that had not been used in the development of the tree. The application of classification trees led to valuable insights with implications for operational quality control processes. Furthermore, classification tree methods were shown to be able to select cases for future quality control processes accurately and efficiently, thereby suggesting that future quality control selection procedures may be completely automated. However, further analyses are needed to establish whether classification trees can be relied on to identify cases that are the most likely to require some adjustment without incurring the potentially costly error of ignoring solutions that are likely to require adjustment. (Contains 10 tables, 7 figures, and 13 references.) (Author/SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A