The bibliometric methodology is an established technique for research evaluation as it offers an objective determination and comparison of research performance. This paper aims to critically assess some standard bibliometric indicators commonly used (based on publication and citation counts) to evaluate academic units, and examine whether there are factors not taken into account that influence evaluation results. Findings suggest that the dissimilarity of academic units (for example regarding their scientific orientation and size in terms of staff number), if not taken into consideration may lead to incorrect evaluation results. This issue becomes even more important when comparing larger academic units, such as universities or colleges. Among the suggested further research is the replication of this study in an expanded time frame and an international context.