In order to use assessment findings for learning improvement, it is essential to improve the validity of these data. When an assessment is distributed across numerous course sections with multiple faculty members acting as assessors, issues arise concerning validity and calibration. To address these concerns, we created a training program to better transmit student learning outcome definitions, provide practice activities aimed at calibration, and promote social moderation among our assessment community. This training, hosted on our university's learning management system, explicitly defines outcomes, establishes common indicators to inform the rating of student proficiency levels, and provides quizzes with sample student work to practice and refine assessment calibration. This training was developed by the assessment team, is given to an assessor shared across all sections, and is modified annually by faculty and program leadership.