NotesFAQContact Us
Search Tips
ERIC Number: ED545236
Record Type: Non-Journal
Publication Date: 2013-Aug
Pages: 47
Abstractor: ERIC
Reference Count: 7
A Validity Study of the NAEP Full Population Estimates
Hedges, Larry V.; Bandeira de Mello, Victor
American Institutes for Research
In early 2001, to support an internal evaluation of the impact of changing exclusion rates on reports of statistically significant gains across states, the National Center for Education Statistics (NCES) sponsored research on imputation procedures of National Assessment of Educational Progress (NAEP) scores for the excluded students and provided adjusted or full population estimates (FPEs) for the 1996 to 2000 NAEP mathematics gains. The same method was subsequently used to produce FPEs for Grades 4 and 8 in reading, writing, mathematics, and science for each year these assessments were administered since 1994. In spite of considerable prior research, NAEP FPEs have not yet achieved operational status, and further validity research has been recommended. The occasion of two special inclusion studies in the 2011 NAEP assessment--designed to determine if students who are excluded from the operational assessment can meaningfully participate if offered one of two special booklets--presented just such an opportunity for additional validity research on the FPEs. The Educational Testing Service (ETS) analyzed the data from these otherwise excluded students, but the analyses were not carried out for the purposes of validating the FPEs. The purpose of this document is to report a study that capitalized on these inclusion studies to obtain further validity evidence about the FPEs and to better inform a possible decision to move toward operational status for the FPE methodology. Fieldwork for two closely related inclusion studies was carried out as part of the 2011 NAEP Mathematics Assessment. The two studies had slightly different student populations and examined different accommodations. One study focused on the use of calculators as an accommodation (in states that permit this accommodation on their state assessments). The other study focused on an inclusion booklet composed of Knowledge and Skill Appropriate (KaSA) blocks. In some states, there were students included in both studies (i.e., some students included because of the calculator accommodation and other students included because of the inclusion blocks). In other states, only the inclusion block was offered because the states did not allow a calculator accommodation on their state assessments. In conclusion, these studies suggest that the calculator block and KaSA booklet accommodations, by themselves, will not change the number of included students enough to have a substantial impact on national parameter estimates. Each study is outlined briefly in this report. Procedures for Calculating Full Population Estimates is appended.
American Institutes for Research. 1000 Thomas Jefferson Street NW, Washington, DC 20007. Tel: 202-403-5000; Fax: 202-403-5001; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: Grade 4; Grade 8; Elementary Education; Middle Schools
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: National Center for Education Statistics (ED); American Institutes for Research
Identifiers - Assessments and Surveys: National Assessment of Educational Progress
IES Funded: Yes
Grant or Contract Numbers: ED-04-CO-0025/0012