This article examines a critical incident during research investigating a new assessment system for on-job learning in carpentry. The system was designed to establish clear relationships between supportive learning environments and purposeful, professional assessment of learners' progress through "naturally occurring evidence" on the building site. However an unfortunate set of circumstances produced an assessment "perfect storm" for a training advisor's workplace visit. It provoked distress and disquiet for all the people involved: the apprentice, the employer, the training advisor/assessor, the moderators, and the researcher. The article frames the incident as a catalyst for critical reflection and shared learning. It argues that certain features in the system, including an on-job assessment community of practice based around "social moderation" of assessment judgements, helped create both the critical incident and the subsequent renegotiation of practice and realignment of relationships.