Many testing programs face the practical challenge of having limited resources to conduct comprehensive standard setting studies. Some researchers have suggested that replicating a group's recommended cut score on a full-length test may be possible by using a subset of the items. However, these studies were based on simulated data. This study describes a standard setting application using two independent panels providing judgments on a 300-item licensure test. Specifically, one panel provided judgments on all 300 items; whereas the second panel made judgments on a randomly-selected subset of 150 items. Both panels also participated in an alternate standard setting method to evaluate panel comparability. Results suggest caution for practitioners considering using subsets of items for standard setting studies. (Contains 4 tables and 1 footnote.)