ERIC Number: ED246104
Record Type: RIE
Publication Date: 1984-Apr
Reference Count: 0
Identification of Items That Are Hard to Rate Accurately Using Angoff's Standard Setting Method.
Melican, Gerald; Thomas, Nancy
Setting standards for the purpose of certification is frequently performed using judgmental techniques such as the Angoff method. This study was performed to identify types of items that judges find hard to rate accurately, that is, types of items on which examinees perform differently than predicted by the judges. Once identified these item types should be the subject of additional training for future panels of judges. Observed difficulty estimates were regressed on the Angoff estimates of a panel of judges for three examinations and items with large residuals were identified. For one examination there were indications that the judges found negatively phrased items hard to rate accurately. For a second examination there were indications that the judges found items involving calculations as hard to rate accurately. In both cases the judges tended to underestimate the difficulty of the items; the examinees found the items harder than the judges predicted. No discernible pattern was found for the third examination. (Author)
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers: Angoff Methods
Note: Paper presented at the Annual Meeting of the American Educational Research Association (68th, New Orleans, LA, April 23-27, 1984).