This study used generalizability theory and other statistical concepts to assess the application of the Angoff method to setting cutoff scores on two professional certification tests. A panel of ten judges gave pre- and post-feedback Angoff probability ratings of items of two forms of a professional certification test, and another panel of nine judges similarly rated and received feedback for the items of two forms of a similar test for more advanced certification. Feedback for a new item consisted of mean judge rating of the item. Feedback for an old item consisted of mean rating plus the item's p-value for a group of examinees who had scored close to the previous cutoff score. Results show that: (1) mean post-feedback ratings were significantly greater than initial ratings for all four test forms; (2) hypothesized greater internal consistency of second ratings was observed only for the first judge group and test; (3) correlations of mean (over judges) first and second ratings were lower for old items than for new items; (4) judges varied widely in the correlation of their first and second ratings; and (5) most judges were strongly influenced by the marginal-group p-values given for old items. (Author/JAZ)