ERIC Number: ED307293
Record Type: RIE
Publication Date: 1989-Mar
Factors Influencing the Degree of Intrajudge Consistency during the Standard Setting Process.
Plake, Barbara S.; And Others
The accuracy of standards obtained from judgmental methods is dependent on the quality of the judgments made by experts throughout the standard setting process. One important dimension of the quality of these judgments is the consistency of the judges' perceptions with item performance of minimally competent candidates. Several interrelated sources of intrajudge consistency can be identified. Variables related to the judges themselves may provide a source of differential consistency of ratings during the standard setting process. Aspects of the items within the examination evaluated in the standard setting process may be an additional contributing source of intrajudge consistency. These two types of factors may interact to yield an additional source of inaccuracy in the judges' ratings. Finally, dimensions of the cutoff study could affect the degree of intrajudge consistency. Strategies to improve intrajudge consistency include: (1) periodic rating; (2) utilizing estimates of minimally competent item performance; (3) analyzing empirical data on item performance; (4) reviewing mathematical models for estimates of item performance; and (5) providing descriptive data on judge groups' ratings. Research is needed to demonstrate the potential power of these various techniques. (SLD)
Publication Type: Speeches/Meeting Papers; Reports - Evaluative
Education Level: N/A
Authoring Institution: N/A
Note: Paper presented the Annual Meeting of the National Council on Measurement in Education (San Francisco, CA, March 28-20, 1989).