In 2001, the U.S. Citizenship and Immigration Services (USCIS) began the process of redesigning the U.S. naturalization tests due to concerns that the current testing procedure may not be sufficiently uniform, and that the test content may not be appropriate. A National Research Council committee issued recommendations, based largely on the…

The psychometric literature is replete with comprehensive discussions of test validity, test validation, and the characteristics of quality assessment programs. The most authoritative source for guidance regarding sound test development and evaluation practices is the Standards for Educational and Psychological Testing. However, the Standards are…

By analyzing the Fisher information allotted to the correct response of a Rasch binary item, Huynh (1994) established the response probability criterion 0.67 (RP67) for standard settings based on bookmarks and item mapping. The purpose of this note is to help clarify the conceptual and psychometric framework of the RP criterion.

A conceptual framework is proposed for a psychometric theory of standard setting. The framework suggests that participants in a standard setting process (panelists) develop an internal, intended standard as a result of training and the participant's background. The goal of a standard setting process is to convert panelists' intended standards to…

In the Spring 2005 issue, Wang, Schnipke, and Witt provided an informative description of the task inventory approach that centered on four functions of job analysis. The discussion included persuasive arguments for making systematic connections between tasks and KSAs. But several other facets of the discussion were much less persuasive. This…

"In his commentary to our paper on the use of knowledge, skill, and ability statements in developing credentialing examinations (Wang, Schnipke, & Witt, 2005 )," Dr. LaDuca set forth his concerns while commending our paper for providing helpful insights into the importance of careful delineation of KSAs. We believe that there is little substantive…

The attractiveness of computer-based tests (CBTs) is due largely to their capability to expand the ways we conduct testing. A relatively unexplored application, however, is actively using the computer to reduce construct-irrelevant variance while a test is being administered. This investigation introduces the effort-monitoring CBT, in which the…

We found moderate correlations among four policy attributes (consistency, specificity, authority, and power), which suggest that in many states, at least in design, standards-based reform is working as advocates imagined--aligned content standards and assessments established, backed up by detailed guidelines and frameworks, incentivized by rewards…

The question addressed in this paper is whether there are signs that students' progression through high school to graduation has been affected by the efforts to increase school accountability, including high-stakes exit exams. If such accountability systems were able to raise student academic performance significantly in elementary and middle…

The No Child Left Behind Act allows states to vary (a) the trajectories they select to move from the baseline percent proficient or above in 2002 to the 100% proficient goal in 2014, (b) the minimum number of students required for reporting of disaggregated subgroup results, and (c) whether or not they will use confidence intervals when…

This study explores the use of multiple measures to enhance the validity and reliability of inferences about school and district effectiveness. Using data from the state of Ohio, a framework for combining measures is applied to examine the individual and collective impact of multiple measures on both the federal AYP designations and state ratings.…

Since 2001, considerations of school reform have been dominated by performance-based accountability. No Child Left Behind (NCLB) has changed the way policymakers and educators talk about education, look at educational performance, and think about educational challenges. Nonetheless, NCLB and the state accountability systems it has spawned have…

In this paper I describe and illustrate the Roussos-Stout (1996) multidimensionality-based DIF analysis paradigm, with emphasis on its implication for the selection of a matching and studied subtest for DIF analyses. Standard DIF practice encourages an exploratory search for matching subtest items based on purely statistical criteria, such as a…

The task inventory approach is commonly used in job analysis for establishing content validity evidence supporting the use and interpretation of licensure and certification examinations. Although the results of a task inventory survey provide job task-related information that can be used as a reliable and valid source for test development, it is…

A note from the Working Group of the Joint Committee on Testing Practices: The "Code of Fair Testing Practices in Education (Code)" prepared by the Joint Committee on Testing Practices (JCTP) has just been revised for the first time since its initial introduction in 1988. The revision of the Code was inspired primarily by the revision of the…