ERIC Number: ED441036
Record Type: Non-Journal
Publication Date: 2000-Apr
Reference Count: N/A
Analysis of Item Ratings for Ensuring the Procedural Validity of the 1998 NAEP Achievement-Levels Setting.
The Achievement-Levels Setting (ALS) process for the National Assessment of Educational Progress (NAEP) resulted in numerical cutscores on the NAEP score scale representing the performance standards for three achievement levels: Basic, Proficient, and Advanced. This paper focuses on an important, but less researched, aspect of the standard setting process using an item-rating approach--the patterns and changes in panelists' item ratings. Instead of analyzing the cutpoints based on item ratings, this paper presents analyses of intrarater consistency in item ratings within and across rounds of ratings, reasonableness of panelists' item ratings, and changes in ratings, in relation to cutpoints, across rounds, using the NAEP Civics examination as an example. For the 1998 achievement level setting process, the NAEP incorporated "Reckase" charts to provide extensive and easy-to-understand feedback regarding the consistency of the panelists' own ratings in the Item Response Theory context. Although the impact of Reckase charts on improving intrarater consistency could not be examined directly, Reckase generally indicated that they found the charts to be useful and informative. The analysis findings summarized in this paper are regarded as evidence of the procedural validity for the 1998 ALS process and the safeguard for the validity of the 1998 ALS outcomes. Generally, panelists were able to improve their item ratings over time. The intervention aimed at improving intrarater consistency seemed to be effective, but it was not dominating in driving panelists' subsequent item ratings. (Contains 3 tables, 4 figures, and 13 references.) (SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Sponsor: ACT, Inc., Iowa City, IA.; National Assessment Governing Board, Washington, DC.
Authoring Institution: N/A
Identifiers - Assessments and Surveys: National Assessment of Educational Progress