ERIC Number: ED302573
Record Type: Non-Journal
Publication Date: 1987-Apr-18
Pages: 14
Abstractor: N/A
ISBN: N/A
ISSN: N/A
EISSN: N/A
Effects of Essay Order on Raters' Score Assignments in a Large-Scale Writing Assessment.
Ferrara, Steven F.
The necessity of controlling the order in which trained essay raters for a statewide writing assessment program receive student essays was studied. The underlying theoretical question concerns possible rater bias caused by raters reading long strings of essays of homogeneous quality; this problem is usually referred to as context effect or contrast effect. The program involves the Maryland Writing Test (MWT), which is administered each year to high school students (grades 9 through 12) as a graduation requirement. Each essay in the test is scored by two raters on a four-point scale using a holistic procedure. A sample of 1,000 MWT essay booklets from over 85,000 student booklets was selected. Raters (23 narrative and 15 explanatory) who participated in the live scoring of the 1986 MWT test program scored 957 pairs of essays. Three ordering schemes were assessed. There were 433 double readings of essays on one topic, and 652 double readings of essays on another topic. Analysis of variance on resulting data indicated that the order in which essays were placed into scoring packets did affect the raters' score assignments, at least for the narrative essay prompt used in the study. The same conclusion cannot be supported for the explanatory prompt. Other results indicate that rater agreement rates and pass-rates do not differ by essay sequence, and that some raters are more reliable scorers than are others within and across sequence conditions. Five data tables are included. (TJH)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A


