In 1991 Pennsylvania began implementation of a direct writing assessment at the sixth-grade and ninth-grade levels. A total of 18,758 sixth graders and 16,575 ninth graders wrote a response to 1 of 9 prompts reflecting 3 modes of writing. A six-point holistic scale was used to score the papers, with two readers scoring each paper. A third reader, or arbitrator, helped resolve any disagreements in the two scores. In all, there were 1,874 arbitrations. This study focused on the reasons for these arbitration decisions, asking whether such factors as the particular prompt, the writer's grade level, or the achievement level of the readers were related to such disagreements. Results indicate that arbitration rates varied by prompt, by mode, and for each of the two grade levels. Approximately 50% of the raters showed a tendency toward giving either high scores or low scores. At one-half of the tables of raters, one individual was found to be involved in at least 40% of the arbitrations. Neither characteristics of the raters nor the achievement level of the raters' students were found to be related to the arbitration statistics computed for each rater. There are 7 tables of study findings and an 11-item list of references. An appendix contains the holistic scoring guide. (SLD)