ERIC Number: ED481664
Record Type: Non-Journal
Publication Date: 2002-Oct
The Effect of Benchmark Selection on the Assessed Quality of Writing.
Popp, Sharon E. Osborn; Ryan, Joseph M.
The purpose of this study was to investigate whether the selection of benchmark writing samples influences the assessment of students' writing quality. More than 300 grade 3 writing samples were scored in two separate rating sessions. Within each scoring session, raters used a different set of benchmark writing samples. Raw ratings were analyzed using multifacet Rasch models. Raw ratings and Rasch parameter estimates were examined and compared for the two sets of ratings. Ratings were also compared to hypothetical performance standards to illustrate the impact of differential benchmark selection. The same writing samples received very different ratings when different benchmark papers were used in scoring despite the uniform rubric. Results imply that assessed quality of writing may depend more on the benchmarks chosen to define the rubric than on the rubric itself. Results confirm the need for continued investigation into sources of construct-irrelevant variance in the design and development of writing assessments and suggest caution in the use and interpretation of large-scale writing assessment scores. (Contains 3 tables, 3 figures, and 12 references.) (Author/SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A