A primary goal of standards-based statewide achievement tests is to classify students into achievement levels that enable valid inferences about student content area knowledge and skill. Explicating how knowledge and skills are expected to differ in complexity in achievement level descriptors, and how that complexity is related to empirical item…

This study examines the potential of the tree diagram, a type of graphic organizer, as an assessment tool to measure students' knowledge structures in statistics education. Students' knowledge structures in statistics have not been sufficiently assessed in statistics, despite their importance. This article first presents the rationale and method…

Accountability in higher education has increased, with more institutions requiring standardized tests. These tests are high stakes for institutions, but low-stakes test for students, who seldom experience consequences for their performance. This study describes how one psychology department improved students' scores on the Psychology Area…

This article presents a study of ethnic Differential Item Functioning (DIF) for 4th-, 7th-, and 10th-grade reading items on a state criterion-referenced achievement test. The tests, administered 1997 to 2001, were composed of multiple-choice and constructed-response items. Item performance by focal groups (i.e., students from Asian/Pacific Island,…

In the USA, trends in educational accountability have driven several models attempting to provide quality data for decision making at the national, state, and local levels, regarding the success of schools in meeting standards for competence. Statistical methods to generate data for such decisions have generally included (a) status models that…

In this article, I specify a conceptual framework for test validity research on content assessments taken by English language learners (ELLs) in U.S. schools in grades K-12. This framework is modeled after one previously delineated by Willingham et al. (1988), which was developed to guide research on students with disabilities. In this framework…

English language learners (ELLs) constitute one of the fastest growing subpopulations of students in the United States. It is important to determine whether the assessments used by states in determining students' proficiencies are valid and fair for ELLs. This study focused on several standards-based assessments in mathematics and science…

This study explores two approaches to directly measuring the quality of instruction: teachers' assignments with student work and focused lesson observations. The technical quality and potential feasibility of these approaches for measuring instruction in large numbers of classrooms are compared within two different content areas (reading…

The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a…

In this article we examine empirical evidence on the criterion, predictive, transfer, and fairness aspects of validity of a large-scale language arts performance assessment, referred to as the Performance Assignment (PA). We use multilevel models to avoid biased inferences that might result from the naturally nested data. Specifically, we examine…

Validation of assessments intended to improve instruction and learning should include evidence of instructional sensitivity. This study investigated the instructional sensitivity of a standards-based ninth-grade performance assessment that required students to write an essay about conflict in a literary work. Before administering the assessment,…

This article reports the technical quality of a measure describing the quality of classroom assignments piloted in the Los Angeles Unified School District's proposed new accountability system. For this study, 181 teachers were sampled from 35 schools selected at random. Participating teachers submitted three language arts assignments with samples…

A number of methods for scoring tests with selected-response (SR) and constructed-response (CR) items are available. The selection of a method depends on the requirements of the program, the particular psychometric model and assumptions employed in the analysis of item and score data, and how scores are to be used. This article compares 3 methods:…

Studied the relationship between Scholastic Assessment Test (SAT) scores and freshman grades based on the records of 77,893 students at a university system. Findings show that SAT II achievement tests are better predictors than the SAT I test, and the predictive value of SAT II tests is less conditioned by socioeconomic status than the SAT I. (SLD)

Studies of the achievement test results for about 490 school districts at grade 4 and about 420 districts at grade 5 show that the error variance of estimates of change at the school level is large enough to interfere with interpretations of annual change estimates. (SLD)