ERIC Number: ED198156
Record Type: Non-Journal
Publication Date: 1980-Jun
Reference Count: N/A
An Empirical Comparison of Judgmental Approaches to Standard Setting Procedures.
Rock, D. A.; And Others
An experiment was designed that varied cutting score procedures, instructions, and types of judges in order to address the following questions concerning the Real Estate Licensing Examination: (1) Will the cutting score levels produced by groups of judges from differing backgrounds (academicians vs. practitioners vs. lawyers) using the same method and instructions be different? (2) Will the agreement between item rating profiles vary across these different groups of judges? (3) Does either agreement across items and/or levels vary systematically by instruction/method? It was found that three out of four groups of judges arrived at significantly higher cutting score levels using the Angoff method than when using the Nedelsky procedure. The Angoff procedure was more effective in setting standards that distinguished the minimally qualified practitioner from the individual with average qualifications. Although the Angoff method demonstrated somewhat higher interjudge agreement with respect to the patterns of item responses the average correlation between item profiles was generally low for both procedures. (Author/GK)
Publication Type: Reports - Research
Education Level: N/A
Authoring Institution: Educational Testing Service, Princeton, NJ.