Explored the relationship between score changes from the Maryland State Performance Assessment Program (1993 -1998) and classroom instruction and assessment, learning and motivation, beliefs about the assessment, and school characteristics. Several factors from each dimension explained a significant amount of variability in school performance over…

Studied whether differential item functioning (DIF) in translated verbal items could be reduced or eliminated by revising these items. Results for six sections of an Israeli college admission test translated from Hebrew to Russian show that revisions can reduce DIF considerably. Discusses costs of the revision process. (SLD)

Conducted a simulation study of differential item functioning (DIF) to compare the power and Type I error rates for two conditions: using an examinee's ability estimate as the conditioning variable with the CATSIB program and either using the regression correction from CATSIB or not. Discusses implications of findings for DIF detection. (SLD)

Examined the effects of test length and sample size on the alternate forms reliability and equating of simulated mathematics tests composed of constructed response items scaled using the two-parameter partial credit model. Results suggest that, to obtain acceptable reliabilities and accurate equated scores, tests should have at least 8 6-point…

Describes examinee-centered method for setting multiple cutscores on a test involving both objective and extended-response items. Judges evaluate a representative sample of examinee performance using a rating scale that is defined in terms of performance standards, and these ratings are linked to examinee's test scores to generate a functional…

Investigated a theoretical framework describing the role of classroom assessment in student effort and achievement using data from four classroom assessments in each of two third-grade classrooms, including interviews with 32 students. In general, the expected positive relationships were found among perceptions of task, self-efficacy, effort, and…

State-level administrator certification requirements were studied with respect to student-assessment expectations, using responses of state certification offices. Only 18 states require some form of student-assessment knowledge and skills, and only Washington state requires the assessment competencies promulgated by the National Policy Board for…

The process used to identify the provisional locations of points that would inform the National Assessment Governing Board in setting achievement levels for the science National Assessment of Educational Progress (NAEP) is described. This process includes identification of panelists for achievement-level setting and the method for converting…

The validity of interpretations of National Assessment of Educational Progress (NAEP) achievement levels is evaluated by focusing on evidence regarding three types of discrepancies: (1) between standards; (2) among descriptions of achievement levels; and (3) between assessments and content standards. All of these discrepancies raise serious…

How summarizing National Assessment of Educational Progress (NAEP) results in terms of a market basket of items would affect achievement-level reporting is discussed. A market basket is a specific set of items one may administer, scores on which constitute a reporting scale. Advantages and limitations of the market-basket approach are discussed.…

Credentialing programs were surveyed to determine the procedures they use to set performance standards on multiple-choice and open-ended assessments. Implications of the various standard-setting approaches for the National Assessment of Educational Progress are discussed, and it is asserted that generalizing from standard-setting in professional…

Lessons that can be drawn from the standard-setting practices of the U.S. military services for the National Assessment of Educational Progress (NAEP) are explored. The main lesson, and distinction, to be learned is that standards must be grounded in real-life outcomes that affect the lives of educators, something not characteristic of the present…

A brief history of developments in the assessment of the reliability of credentialing examinations is presented, and some new results are outlined that highlight the interactions among scoring, standard setting, and the reliability and validity of pass-fail decisions. Decision consistency is an important concept in evaluating credentialing…

Causes of local item dependence in a large-scale performance assessment were studied using data from the Maryland School Performance Assessment Program. Contextual characteristics (content and response requirements) were identified to differentiate locally independent and dependent item clusters. Hypothesized explanations are offered for high…

The usefulness of small-scale interview studies to explore the validity of science achievement tests in several formats was studied with 141 high school students and 49 fifth and sixth graders. Results, which provided evidence of test validity, also support the use of small-scale qualitative interviews in test validation. (SLD)