The use of students' "consumer feedback" to assess faculty behavior and improve the process of medical education is a significant challenge. We used quantitative Rasch measurement to analyze pre-categorized student comments listed by 385 graduating medical students. We found that students differed little with respect to the number of comments they provided and that their comments indeed form a probabilistic Rasch hierarchy. However, different hierarchies were found across medical departments and faculty. An analysis of these interactions provides valuable, detailed, and quantitative information that can augment qualitative research approaches. In addition, we suggest how the Rasch scaling of student comments can assist researchers in the design and implementation of new faculty evaluation instruments. Finally, the interactions between student and department identified a subset of behaviors that appear to guide and possibly elicit students' comments.