Classroom observation protocols, in which observers rate multiple dimensions of teaching according to established protocols (either live in the classroom, or post-hoc from lesson videos), are increasingly being used in both research and policy contexts. However, scores generated from these protocols have many sources of error. Day to day variation in classroom activities, and variation in curricula and context across different classrooms a teacher may teach mean that a small sample of lessons could vary widely on scores even when scored by a common rater. Variation among raters and changes in raters' use of rubric score scale over time also contribute to error. The purpose of this study is to use scores from video observations from multiple teaching observation protocols administered as part of the Understanding Teaching Quality (UTQ) study to investigate and understand the nature of rater variation due to rating severity, including overall time trends and variations among raters. The UTQ study included 458 teachers teaching mathematics (n=231) or English language arts (n=227) to sixth, seventh, and eighth graders in middle schools in three large school systems from the same metropolitan region in the southeastern United States. UTQ collected data over two school years with roughly half of the teachers participating in each year. For each classroom, two lessons were video recorded and then scored using three observation protocols: Classroom Assessment Scoring System, Secondary, CLASS-S (Pianta, Hamre, Haynes, Mintz, & LaParo, 2007) and Danielson's Framework for Teaching, FFT, (Danielson, 2007) and either the Protocol for Language Arts Teaching Observations, PLATO (Grossman et al., 2010), for English language arts classes, or the Mathematical Quality of Instruction, MQI (Hill et al., 2008), for mathematics classes. The authors have not yet fit the augmented variance decomposition model to these data. They present preliminary findings based on the CLASS-S protocol scores. The descriptive analyses suggest time trends that vary by domain and by rater, suggesting that scoring time is consequential. Tables and figures are appended.