NotesFAQContact Us
Search Tips
ERIC Number: ED523966
Record Type: Non-Journal
Publication Date: 2010-Mar
Pages: 72
Abstractor: As Provided
Reference Count: 30
Studies of a Latent Class Signal Detection Model for Constructed Response Scoring II: Incomplete and Hierarchical Designs. Research Report. ETS RR-10-08
DeCarlo, Lawrence T.
Educational Testing Service
A basic consideration in large-scale assessments that use constructed response (CR) items, such as essays, is how to allocate the essays to the raters that score them. Designs that are used in practice are incomplete, in that each essay is scored by only a subset of the raters, and also unbalanced, in that the number of essays scored by each rater differs across the raters. In addition, all of the possible rater pairs may not be used. The present study examines the effects of these factors on parameter recovery and classification accuracy using simulations of a latent class model based on signal detection theory (SDT). Many tests also include more than one CR item, which introduces a nested or hierarchical structure into the design, in that raters are nested within essays (i.e., there are multiple raters per essay) and essays are nested within examinees (i.e., each examinee provides two or more essays). A hierarchical rater model (HRM) has previously been developed to recognize the nested structure. A version of the HRM that incorporates a latent class signal detection model in the first level, referred to as the HRM-SDT model, is presented. Parameter recovery in the HRM-SDT model is examined in simulations. The model is applied to data from several ETS tests. Appended are: (1) Balanced and Unbalanced Designs Used in the Simulations; (2) Parameter Estimates, Bias, Percent Bias, and MSE; (3) Evaluation of the Estimated Standard Errors for d and the Latent Class Sizes; and (4) Evaluation of Parameter Estimates: HRM-SDT Model. (Contains 24 tables and 2 figures.)
Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site:
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Educational Testing Service