ERIC Number: ED346113
Record Type: RIE
Publication Date: 1992-Apr
Reference Count: N/A
A Study of Arbitrations in Pennsylvania's Writing Assessment.
Masters, James R.
In 1991 Pennsylvania began implementation of a direct writing assessment at the sixth-grade and ninth-grade levels. A total of 18,758 sixth graders and 16,575 ninth graders wrote a response to 1 of 9 prompts reflecting 3 modes of writing. A six-point holistic scale was used to score the papers, with two readers scoring each paper. A third reader, or arbitrator, helped resolve any disagreements in the two scores. In all, there were 1,874 arbitrations. This study focused on the reasons for these arbitration decisions, asking whether such factors as the particular prompt, the writer's grade level, or the achievement level of the readers were related to such disagreements. Results indicate that arbitration rates varied by prompt, by mode, and for each of the two grade levels. Approximately 50% of the raters showed a tendency toward giving either high scores or low scores. At one-half of the tables of raters, one individual was found to be involved in at least 40% of the arbitrations. Neither characteristics of the raters nor the achievement level of the raters' students were found to be related to the arbitration statistics computed for each rater. There are 7 tables of study findings and an 11-item list of references. An appendix contains the holistic scoring guide. (SLD)
Descriptors: Academic Achievement, Arbitration, Decision Making, Educational Assessment, Essay Tests, Evaluation Problems, Grade 6, Grade 9, High Schools, Holistic Evaluation, Intermediate Grades, Interrater Reliability, Performance Based Assessment, Problem Solving, Prompting, Scoring, State Programs, Student Evaluation, Testing Programs, Writing Evaluation
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Note: Paper presented at the Annual Meeting of the National Council on Measurement in Education (San Francisco, CA, April 1992).