In this article, we highlight the need for a precisely defined construct in score-based validation and discuss the contribution of cognitive theories to accurately and comprehensively defining the construct. We propose a framework for integrating cognitively based theoretical and empirical evidence to specify and evaluate the construct. We apply…

This article develops a validity argument approach for use on observation protocols currently used to assess teacher quality for high-stakes personnel and professional development decisions. After defining the teaching quality domain, we articulate an interpretive argument for observation protocols. To illustrate the types of evidence that might…

Student-teacher interactions are dynamic relationships that change and evolve over the course of a school year. Measuring classroom quality through observations that focus on these interactions presents challenges when observations are conducted throughout the school year. Variability in observed scores could reflect true changes in the quality of…

In this article, I specify a conceptual framework for test validity research on content assessments taken by English language learners (ELLs) in U.S. schools in grades K-12. This framework is modeled after one previously delineated by Willingham et al. (1988), which was developed to guide research on students with disabilities. In this framework…

The purpose of this article is to describe and demonstrate a three-step process of using differential distractor functioning (DDF) in a post hoc analysis to understand sources of differential item functioning (DIF) in multiple-choice testing. The process is demonstrated on two multiple-choice tests that used complex alternatives (e.g., "No…

This paper developed a Democratic Classroom Survey to measure students' perceived democratic environment of the classroom. Perceived democratic environment is one of the most important variables for understanding classroom activity and indeed any type of group activity, but actually measuring perceptions in an objective manner has been…

The No Child Left Behind Act of 2001 requires all states to assess the English proficiency of English language learners each school year. Under Title I and Title III of No Child Left Behind, states are required to measure the annual growth of students' English language development in reading, listening, writing, and speaking and in comprehension…

This article discusses the status of existing English language proficiency (ELP) tests, and compares the content coverage and psychometric characteristics of ELP assessments that existed prior to the implementation of the No Child Left Behind Act (NCLB) with those developed after NCLB Title III guidelines were introduced. The article argues that…

Over the previous two decades, the era of accountability has amplified efforts to measure educational effectiveness more than Edward Thorndike, the father of educational measurement, likely would have imagined. Expressly, the measurement structure for evaluating educational effectiveness continues to rely increasingly on one sole…

This article describes the findings of studies conducted on a large-scale, classroom-based performance assessment of literacy for the early grades designed to provide information that is useful for reporting, as well as teaching. Technical studies found the assessment to be a promising instrument that is reliable and valid. Follow-up studies of…

This article describes the history, evidence warrants, and evolution of the Center for Research on Evaluation, Standards, and Student Testing's (CRESST) model-based assessments. It considers alternative interpretations of scientific or practical models and illustrates how model-based assessment addresses both definitions. The components of the…

In this article we describe the development, analysis, and interpretation of a novel item format we call Ordered Multiple-Choice (OMC). A unique feature of OMC items is that they are linked to a model of student cognitive development for the construct being measured. Each of the possible answer choices in an OMC item is linked to developmental…

This article reports findings on growth in 3 science teachers' expertise with interpretation of student work over 1 year of participation in a program. The program was designed to strengthen classroom assessment. Using a framework for classroom assessment expertise, we analyze patterns of teacher learning, and the roles of the professional program…

Value-added measures can be used to allocate funding to schools, to identify those institutions in need of special attention and to underpin government guidance on targets. In England, there has been a tendency to include in these measures an ever-greater number of contextualising variables and to develop ever-more complex models that encourage…

In this paper we described how we mixed research approaches in a Randomized Control Trial (RCT) of a school principal professional development program. Using examples from our study we illustrate how combining qualitative and quantitative data can address some key challenges from validating instruments and measures of mediator variables to…