ERIC Number: EJ941586
Record Type: Journal
Publication Date: 2011-Oct
Pages: 16
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1075-2935
EISSN: N/A
The Accuracy of Performance Task Scores after Resolution of Rater Disagreement: A Monte Carlo Study
Penny, James A.; Johnson, Robert L.
Assessing Writing, v16 n4 p221-236 Oct 2011
When multiple raters score a writing sample, on occasion they will award discrepant scores. To report a single score to the examinee, some method of resolving those differences must be applied to the ratings before an operational score can be reported. Several forms of resolving score discrepancies have been described in the literature. Initial studies of the various methods, however, have demonstrated that decisions about student performance may differ depending on the resolution method applied. Thus, studies are needed to investigate the quality of the scores associated with each model. To study score quality associated with each model, we conducted a Monte Carlo study and varied the factors associated with scoring and resolution to determine the conditions under which a particular resolution method might be superior. (Contains 3 figures and 4 tables.)
Descriptors: Monte Carlo Methods, Scores, Academic Achievement, Models, Scoring, Measurement, Writing Skills, Writing (Composition), Writing Processes, Work Sample Tests
Elsevier. 6277 Sea Harbor Drive, Orlando, FL 32887-4800. Tel: 877-839-7126; Tel: 407-345-4020; Fax: 407-363-1354; e-mail: usjcs@elsevier.com; Web site: http://www.elsevier.com
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A