ERIC Number: ED307306
Record Type: Non-Journal
Publication Date: 1989-Mar
Reference Count: N/A
Selection of Judges for Standard Setting: What Kinds? How Many?
Jaeger, Richard M.
Criteria for the selection of judges (evaluators) for setting item-based standards involved in tests for which cutting scores must be established are investigated. Focus is on cases in which test standards are based on specialists' judgments concerning the difficulty of test items in tests used to determine who will be awarded a diploma, admitted to a program, certified as competent, or licensed to practice. The literature on the nature of expertise describe the qualifications that are required of judges assigned to apply item-based standard-setting procedures. Qualifications for judges of items for tests used to select applicants to teacher education programs, grant initial teacher certificates, and grant high school diplomas are briefly outlined. Post hoc analyses of judges are preferable to competency testing of judges. An item response model outlined by Van der Linden (1982) and an alternative approach involving interrater comparisons outlined by Jaeger (1988) are offered as possible means of post hoc analysis. Problems associated with determination of a sufficient number of judges, including issues involving standard errors of mean and measurement, are outlined. In general, judges should be selected through procedures that permit generalization of their collective recommendations to well-defined populations. The numbers of judges selected should be sufficient to provide precise estimation of the standard that would be recommended by an entire population of judges. (TJH)
Descriptors: College Entrance Examinations, Cutting Scores, Difficulty Level, Estimation (Mathematics), Evaluation Criteria, Evaluators, Graduation Requirements, High School Students, Higher Education, Interrater Reliability, Latent Trait Theory, Licensing Examinations (Professions), Minimum Competency Testing, Qualifications, Secondary Education, Selection, Standard Setting (Scoring), Standardized Tests, Teacher Certification, Teacher Education, Test Items
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A