This study examined the use of measures of student learning computed using end-of-year assessments (distal measures) versus measures of student learning associated with a single lesson (proximal measures) as criterion scores for the validity of observations of teachers' pedagogical skills. The validity coefficients computed using distal measures were significantly lower than the validity coefficient computed using proximal measures. Assumptions underlying the current emphasis on distal measures were challenged. Possible ways to generate more proximal measures were explored.