Institutional assessment of student learning objectives has become a fact-of-life in American higher education and the Association of American Colleges and Universities' (AAC&U) VALUE Rubrics have become a widely adopted evaluation and scoring tool for student work. As faculty from a variety of disciplines, some less familiar with the psychometric literature, are drawn into assessment roles, it is important to point out two easily made but serious errors in what might appear to be one of the more straightforward assessments of measurement quality--interrater reliability. The first error which can occur when a third rater is brought in to adjudicate a discrepancy in the scores reported by an initial two raters has been well-documented in the literature but never before illustrated with AAC&U rubrics. The second error is to cease training before the raters have demonstrated a satisfactory level of interrater reliability. This research note describes an actual case study in which the interrater reliability of the AAC&U rubrics was incorrectly reported and when correctly reported found to be inadequate. The note concludes with recommendations for the correct measurement of interrater reliability.