ERIC Number: ED345784
Record Type: RIE
Publication Date: 1991-Nov
Judgments of Placement Writing Samples at Golden West College: An Evaluation of Inter-Rater Reliability.
At Golden College (California), student writing samples are holistically scored by pairs of judges on a six-point scale. Judges are allowed to use plus and minus figures, thus converting the integer scale to a decimal scale of evaluation. In 1991, 499 writing samples written as part of the placement testing process for students in the Coast Community College District's SOAR program were analyzed for reliable scoring on the part of the judges. Two procedures were used to assess the extent to which judgments of writing samples were consistent. The first method entailed calculating the difference between the two ratings of all writing samples. For 34.4% of the pairs, the ratings from independent judges were identical. Ratings differed by one-third of a point in 30.9% of the cases, by two-thirds of a point in 15.4% of the cases, and by exactly one point in 13.4% of the cases. Overall, ratings differed by one point or less in 94.1% of the cases, exceeding the minimal standard of 90% recommended by the California community colleges (CCC). The second procedure used was a Pearson correlation coefficient which assessed the relationship between paired ratings for the writing samples. The correlation between ratings was moderately strong, and positive (r=.76), exceeding the minimal standard of .75 recommended by the CCC. The data supported the hypothesis that the ratings from independent judges were made in a consistent manner. Two references and an appendix of related data are attached. (JMC)
Publication Type: Reports - Evaluative; Numerical/Quantitative Data
Education Level: N/A
Authoring Institution: Golden West Coll., Huntington Beach, CA.