NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ941586
Record Type: Journal
Publication Date: 2011-Oct
Pages: 16
Abstractor: As Provided
Reference Count: 0
ISSN: ISSN-1075-2935
The Accuracy of Performance Task Scores after Resolution of Rater Disagreement: A Monte Carlo Study
Penny, James A.; Johnson, Robert L.
Assessing Writing, v16 n4 p221-236 Oct 2011
When multiple raters score a writing sample, on occasion they will award discrepant scores. To report a single score to the examinee, some method of resolving those differences must be applied to the ratings before an operational score can be reported. Several forms of resolving score discrepancies have been described in the literature. Initial studies of the various methods, however, have demonstrated that decisions about student performance may differ depending on the resolution method applied. Thus, studies are needed to investigate the quality of the scores associated with each model. To study score quality associated with each model, we conducted a Monte Carlo study and varied the factors associated with scoring and resolution to determine the conditions under which a particular resolution method might be superior. (Contains 3 figures and 4 tables.)
Elsevier. 6277 Sea Harbor Drive, Orlando, FL 32887-4800. Tel: 877-839-7126; Tel: 407-345-4020; Fax: 407-363-1354; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A