A rating scale was developed for the evaluation of behaviors and skills required of medical students. From 400 items based on various role-defining studies, 92 were selected. The project staff randomly assigned 46 items to each of two alternate forms. Seven medical schools were identified as sites for pilot testing. Medical students who agreed to participate were asked to rate themselves and to agree to be rated by two or more resident and faculty supervisors. Ratings were distributed across the four response options available (seldom, sometimes, usually, consistently) on most of the 92 items. Raters were not reluctant to use the more negative responses at times. Very few students used the fifth option (don't know), although physicians used it rather freely. There was a significant inverse relationship between the frequency of use of the don't know option and the number of hours of contact with a student. Interrater reliability ranged from moderate to substantial agreement. The staff had hypothesized that residents would have more extended contact with students than faculty, be more knowledgeable about student performance, and therefore provide more reliable ratings. The data supported the first two hypotheses, but not the third. (Author/GDC)