There is currently much controversy in the student evaluations of teaching, employed by many universities world-wide. The administrators see these as a valuable means to measure a teacher's effectiveness and, hence, potential for advancement, while much of the research suggests that these may be unreliable proxies for how effective the student learning actually is. Whether or not the evaluations in their current format are eventually deemed as reliable, one issue of crucial importance is the class size, which must be correctly weighted in determining the teacher's overall effectiveness. Here, through a case study, we demonstrate the possible pitfalls in using the scores from a small number of student responses.