ERIC Number: ED420674
Record Type: Non-Journal
Publication Date: 1998-Apr
Reference Count: N/A
Analysis of Methods for Collecting Test-based Judgments.
Reckase, Mark D.
Standard setting is a fairly widespread activity in educational and psychological measurement, but there is no formal psychometric theory to guide the development of standard setting methodology. This paper presents a conceptual framework for such a psychometric theory and uses the conceptual framework to analyze a number of methods for setting standards through judges' interactions with test materials. The various standard-setting methods that have been used or are being considered for setting National Assessment of Educational Progress standards are considered. The model presented indicates that the standard is set by the agency that calls for the standard, and the task of the judges is to translate the agency's description of the standard, the task definition, to a numerical value on the reported score scale. The translation process is influenced by several features of the standard setting process, including the creation of content descriptions and the selection of the standard setting methodology. Several standard setting methods are evaluated to determine the likelihood that the judges' ratings could be used to recover the standard in a statistically unbiased way with a reasonably small standard error. Sources of variation in estimates of standards were considered, including the quality of translation of task definitions to content descriptions, the level of understanding by judges of content descriptions and item characteristics, and the amount of information acquired from the judges. Future work will emphasize formalizing concepts and developing analytic models of the standard setting process that can be used to guide data-based evaluations of the statistical quality of standards. (Contains 1 table, 1 figure, and 24 references.) (SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: National Assessment of Educational Progress