We compared three assessor types (self-assessors, peer-assessors, and teacher assessors) to determine whether they differed in the levels of severity they exercised when rating essays. We analyzed the ratings of 194 assessors who evaluated 188 essays that students enrolled in two state-run universities in Iran wrote. The assessors employed a…

In this paper, we provide an overview of psychometric procedures and guidelines Educational Testing Service (ETS) uses to evaluate automated essay scoring for operational use. We briefly describe the e-rater system, the procedures and criteria used to evaluate e-rater, implications for a range of potential uses of e-rater, and directions for…

This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters…

The decision-making behaviors of 8 raters when scoring 39 persuasive and 39 narrative essays written by second language learners were examined, first using Rasch analysis and then, through think aloud protocols. Results based on Rasch analysis and think aloud protocols recorded by raters as they were scoring holistically and analytically suggested…

Mixed quantitative and qualitative methods were used to assess student attitudes toward the assessment criteria for higher-order critical and analytical thinking skills in writing-intensive curricula in first-year, general education courses. The courses varied in emphasis on critical thinking in the criteria used to assess writing. The analysis is…

This experimental project investigated the reliability and validity of rubrics in assessment of students' written responses to a social science "writing prompt". The participants were asked to grade one of the two samples of writing assuming it was written by a graduate student. In fact both samples were prepared by the authors. The first sample…

Current concerns about academic plagiarism in student writing assume qualitative and quantitative differences in the writing of students for whom English is a first language (EL1) and English is a second language (EL2), but lack precision in measuring those differences. I examined the citation practices of EL1 and EL2 students in a timed writing…

Writing is a highly valued skill that is often neglected in the classroom; one reason is that teachers often do not receive adequate training in writing assessment and instruction. Teachers, particularly preservice teachers, need practice making detailed assessments of student writing and to build their confidence for assessing student writing,…

Test-based accountability programs are designed to promote improved standards of teaching and learning within the systems of education that they are connected to. Brenan [Brenan, A. L. (2006). "Perspectives on the evolution and future of educational measurement." In: Robert (Ed.), "Educational measurement" (4th ed., pp. 1-16). Westport, CT:…

Despite the debate among writing researchers about its viability as a pedagogical tool in writing instruction [e.g., Helms-Park, R., & Stapleton, P. (2003). "Questioning the importance of individualized voice in undergraduate L2 argumentative writing: An empirical study with pedagogical implications." "Journal of Second Language Writing," 12 (3),…

Classroom assessment of writing is considered from an anthropological perspective as practitioners' tool use. Pan Canadian data from a 2002 English teacher questionnaire (N = 4070) about self-reported assessment practices were analyzed in terms of tool choice and use by secondary teachers of different experience and qualification levels. Four…

In this article, the authors argue that evaluation courses grounded in assessment theory and situated within a cultural context of actual workplace practices enhances student learning. Giving students the tools that assessors, both formal and informal, use helps them learn to anticipate and generate those tools for writing situations they will…

Inoue [Inoue, A. B. (2005). "Community-based assessment pedagogy." "Assessing Writing: An International Journal, 3", 208-238] sets up a radically experimental writing class as a kind of laboratory of assessment. He seeks to avoid the standard situation where a teacher unilaterally assesses and grades student writing, using only his or her own…

The availability of multi-media technologies in education has made the option of independent learning increasingly attractive. Whilst independent learning presents learners with a more flexible learning context, it also presents new challenges in assessment in that the onus is placed upon the learners themselves to monitor and evaluate their own…

A pilot study was conducted to evaluate and improve the rating procedure proposed for use in a research effort designed to assess the essay writing ability of college sophomores. Generalizability theory and the Many-Facet Rasch Model were each used to (a) estimate potential sources of error in the rating, (b) to obtain reliability estimates, and…