This study investigated ways to improve the appraisal of vocal performances within a national academy of music. Since a criterion-based assessment framework had already been adopted, the conceptual foundation of an assessment rubric was used as a guide in an action research project. The group of teachers involved wanted to explore thinking underpinning their individual judgements in order to make explicit the criteria and standards against which assessment and grading were made. The purpose was to reduce uncertainty and increase transparency to make assessment more predictable, reliable and fair. It was also hoped that the rubric would help to avoid disagreements since criteria were communicated openly. However, disagreements took place over the level of detail in criteria and what would be a sensible way of making use of explicit criteria. The study concludes that while the rubric carries a great potential in aligning judgements, it cannot fully replace a professional holistic appraisal.