ERIC Number: ED175920
Record Type: RIE
Publication Date: 1979-Apr
Reference Count: 0
A Comparison of Two Cutting Score Procedures Using Generalizability Theory. ACT Technical Bulletin No. 33.
Brennan, Robert L,; Lockwood, Robert E.
Procedures for determining cutting scores have been proposed by Angoff and by Nedelsky. Nedelsky's approach requires that a rater examine each distractor within a test item to determine the probability of a minimally competent examinee answering correctly; whereas Angoff uses a judgment based on the whole item, rather than each of its components. The reliability of these approaches depends upon the extent to which raters agree in their judgments. Generalizability theory was used to quantify the magnitude of error variance in each procedure; to compare data resulting from each procedure; and to examine the impact of rater disagreement on test reliability. Five subject experts rated the probability of answering correctly for a total of 126 four-option items in a health-related area. Both procedures were used by the same raters. Cutting score was assumed to be the observed mean (probability) over raters and items. In this sense, the expected variability of the observed mean was error variance attributable to the procedure used. Results indicated that both the cutting scores and their expected variance were considerably different for the two procedures, and suggested that differences between the procedures may be of greater consequence than their apparent similarities. (GDC)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: American Coll. Testing Program, Iowa City, IA. Research and Development Div.
Identifiers: Angoff Method; Generalizability Theroy; Interrater Reliability; Nedelsky Method
Note: Paper presented at the Annual Meeting of the National Council on Measurement in Education (San Francisco, California, April 9-11, 1979)