When classroom observations are increasingly used for accountability and evaluation purposes, a deeper understanding of the psychometric properties of such measurement tools is needed. The present study took a unique approach to examine the psychometric properties of a commonly used classroom observation measure by testing the reliability of indicators for higher-order constructs (i.e. dimensions). We investigated the reliability of indicator ratings of the Classroom Assessment Scoring System (CLASS) Pre-K instrument in Finnish kindergarten and first grade classrooms. Twenty-one observer pairs rated 838 segments identified from the 413 lessons of 48 teachers. Variance components models were specified to investigate variance proportions of each indicator and dimension. The results showed that most observer disagreement was found for the instructional support domain. Observers disagreed relatively more depending on the teacher they observed. There is a clear need for additional understanding on how observers process information on the complex elements of classroom interaction in order to improve training programmes and the reliability and accuracy of the assessment procedure.