This study examined the effects of certain item selection methods on the classification accuracy and classification consistency of criterion-referenced instruments. Three item response data sets, representing varying situations of instructional effectiveness, were simulated. Five methods of item selection were then applied to each data set for the purpose of selecting a best subset of items. Parallel forms of an instrument were then constructed and two measures of test quality obtained: the proportion of correct classifications, and the mean split-half coefficient of agreement. Results indicated that methods yielding the best accuracy and consistency varied across the situations of instructional effectiveness. When a "fair amount" of variability is present in the distribution of test scores, as was the case in Situation 3, the point-biserial correlation method was superior to the other methods. However, in true mastery learning situations, approximated in Situation 2, the Cox and Vargas method of item selection is to be recommended. Finally, in those situations where instruction is relatively ineffective, as was the case with Situation 1, the Brennan and the phi coefficient methods tended to yield the best instruments. (Author/CTM)