ERIC Number: ED237550
Record Type: Non-Journal
Publication Date: 1983-Nov
Reference Count: N/A
Sources of Error in an Angoff Type Standard Setting Process.
McLean, James E.; Lockwood, Robert E.
The sources of variability in the Angoff standard-setting procedure, when applied to the Alabama High School Graduation Examination (AHSGE), were examined. The sources of variability examined are judges, rounds (replications), competencies (items), and interactions among these three sources. After training, the judges were given a statement of a competency, the item specifications for measuring that competency, and a number of sample items based on those specifications. The judges were then asked to estimate the number of items out of 100, similar to the sample items, that they expected a minimally competent student should answer correctly. The initial difficulty estimates (p-values) comprised round one, and the estimates after discussing these p-values comprised round two. The results from the ratings were analyzed using factor analyses of the three sources of variability. The analysis was repeated for the three subject areas of the AHSGE: reading, mathematics, and language. In general, the judge by competency interaction accounted for the largest portion of variance. It seems, therefore, that providing the judges with the actual difficulty levels of items has little impact on the standard setting process. (PN)
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Alabama High School Graduation Examination