Research Findings: Given the variable nature of early childhood settings, practitioners and researchers need better guidance on what conditions influence observations conducted within early childhood settings (National Research Council, 2008). Using 230 observations from 23 three- and four-year-old children, we conducted a Generalizability study to determine what factors influence observation scores of the Individualized Classroom Assessment Scoring System (inCLASS). Results account for a minimal portion of child-level variance across the three inCLASS domains. However, when more contextual factors were analyzed, child-level variance increased as error variance decreased. This resulted in scores that were more reliable. Practice or Policy: We provide recommendations for both future uses of the inCLASS as well as other observational instruments used in research.