Evaluating and improving instructional quality is important in pre-service teacher education, given it is a crucial factor for students' learning gains. This process is complex and involves multiple classroom events with various interpretations. Criteria for instructional quality are rarely applied systematically in teacher education, leading to divergent evaluations and hindering comparable development. Comparability has rarely been researched in science pre-service teacher education. Therefore, we compared evaluations from 17 chemistry-specific advisors and 17 science pre-service teachers with regard to their choice of criteria, their respective rating and grading of a videotaped chemistry lesson, in Germany. Their evaluations were compared using the Science Education Perspectives (SEP) framework for instructional quality. Although advisors can be considered evaluation experts, our findings show differences in the choice of criteria, ratings, and how the lesson was graded within and between both groups. Pre-service teachers focused more on aspects of classroom and time management, and gave higher average ratings, whereas their advisors focused more on cognitive activation. Overall, 16 different criteria were used by the majority of participants. These criteria show a strong science-specific focus. Our findings have implications for science pre-service teacher education, showing a need for a common approach in evaluations, with extended observation periods.