Procedures for determining cutting scores have been proposed by Angoff and by Nedelsky. Nedelsky's approach requires that a rater examine each distractor within a test item to determine the probability of a minimally competent examinee answering correctly; whereas Angoff uses a judgment based on the whole item, rather than each of its components. The reliability of these approaches depends upon the extent to which raters agree in their judgments. Generalizability theory was used to quantify the magnitude of error variance in each procedure; to compare data resulting from each procedure; and to examine the impact of rater disagreement on test reliability. Five subject experts rated the probability of answering correctly for a total of 126 four-option items in a health-related area. Both procedures were used by the same raters. Cutting score was assumed to be the observed mean (probability) over raters and items. In this sense, the expected variability of the observed mean was error variance attributable to the procedure used. Results indicated that both the cutting scores and their expected variance were considerably different for the two procedures, and suggested that differences between the procedures may be of greater consequence than their apparent similarities. (GDC)