NotesFAQContact Us
Search Tips
ERIC Number: ED512708
Record Type: Non-Journal
Publication Date: 2010
Pages: 11
Abstractor: ERIC
Reference Count: 11
Assessing the Conditional Reliability of State Assessments
May, Henry; Cole, Russell; Haimson, Josh; Perez-Johnson, Irma
Society for Research on Educational Effectiveness
The purpose of this study is to provide empirical benchmarks of the conditional reliabilities of state tests for samples of the student population defined by ability level. Given that many educational interventions are targeted for samples of low performing students, schools, or districts, the primary goal of this research is to determine how severe the attenuation in conditional reliability is, depending on the test, subject matter, and average ability level of the students in the research sample. This objective is addressed, along with implications for future research, in research questions 1 and 2 below: (1) What are the conditional reliabilities (defined as the median and 95% CI for pretest-posttest correlations) of given state tests for subsets of the population defined by achievement level?; and (2) What are the implications for power analysis when state assessments are the outcomes of interest for samples of students taken from the tails of the achievement distribution? The results of research question 1 will be used to explore the implications of conditional reliability in the prospective power analysis for a study using state assessments as outcomes. This study demonstrates that the reliability of these assessments may be compromised when low and/or high ability segments of the student population are to be sampled for the intervention. The simulation results indicate that if a study were to be performed on a sample of low performing students (students 2 standard deviations below the mean), the use of a state test would be far less appropriate than if the intervention was focused on students whose ability levels were closer to the average ability level of the population as a whole. These findings have important implications for prospective power analysis and the precision of impact estimates for randomized trial designs. (Contains 1 table and 3 figures.)
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: Elementary Secondary Education
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: Society for Research on Educational Effectiveness (SREE)