Observations seem particularly susceptible to rater error due to the level of subjectivity involved in assessment. Thus, the present paper aims to investigate: (1) inter-rater agreement (IRA) using the Classroom Assessment Scoring System -- Secondary version (CLASS-S) and (2) the CLASS-S factor structure in a Norwegian context. Inter-rater agreement analysis comprise percent within one (PWO). Confirmatory factor analysis are performed to analyze the factor structure. Certified trainers scored videotaped observations. The study indicates that an acceptable IRA can be achieved in a Norwegian setting, and it reports promising results regarding the structural validity of the CLASS-S.