ERIC Number: ED270484
Record Type: RIE
Publication Date: 1986-Apr
Reference Count: 0
Essay Reliability: Form and Meaning.
This study is an attempt at a cohesive characterization of the concept of essay reliability. As such, it takes as a basic premise that previous and current practices in reporting reliability estimates for essay tests have certain shortcomings. The study provides an analysis of these shortcomings--partly to encourage a fuller understanding of the concept of reliability as applied to essay testing, and partly to build the case that the framework of generalizability theory offers a much more satisfactory way of characterizing the concept. The study applies generalizability theory to existing research to illustrate that this approach is an improvement over the usual methods of estimating essay reliability. The paper also argues that the classical approach to reliability has lead to a preoccupation with inter-marker agreement which in turn has led to a formulation of the "reliability problem" that makes it not susceptible of solution. Conceptual grounds for tolerating inter-marker disagreement are advanced and the paper discusses conditions under which this may be so. However, the paper explains how generalizability theory remains an appropriate framework for estimating the reliability of essay scores whatever assumptions one chooses to make regarding inter-marker consistency. (Author)
Publication Type: Speeches/Meeting Papers; Reports - Evaluative
Education Level: N/A
Authoring Institution: N/A
Note: Paper presented at the Annual Meeting of the American Educational Research Association (70th, San Francisco, CA, April 16-20, 1986).