ERIC Number: ED486654
Record Type: Non-Journal
Publication Date: 2005-Nov
Using Covariates to Improve Precision: Empirical Guidance for Studies That Randomize Schools to Measure the Impacts of Educational Interventions. MDRC Working Papers on Research Methodology
Bloom, Howard S.; Richburg-Hayes, Lashawn; Black, Alison Rebeck
This paper examines how controlling statistically for baseline covariates (especially pretests) improves the precision of studies that randomize schools to measure the impacts of educational interventions on student achievement. Part I of the paper introduces the concepts, issues, and options involved. Parts II and III present empirical findings that illustrate how precision is influenced by a wide range of different covariates. These findings were based on multiple years of individual data for student test scores in reading and math from five urban school districts. They represent grades three and five for elementary schools, grade eight for middle schools, and grade ten for high schools. Part IV of the paper compares its results to those of previous research, presents an approach for quantifying uncertainty about its results, and considers what further research is needed. Findings indicate that: (1) pretests can reduce the number of randomized schools required for a given level of precision to about one-half of what would be needed otherwise for elementary schools, one-fifth for middle schools, and one-tenth for high schools; (2) aggregate school-level pretests are as effective in this regard as are individual student-level pretests; (3) the precision-enhancing power of pretests declines somewhat, but not much, as the number of years between the pretest and post-tests increases; (4) the precision enhancing power of pretests for multiple baseline years is only slightly greater than that for a single baseline year; and (5) the precision-enhancing power of pretests is substantial, even when the pretest differs from the post-test. Appended is: Additional Detailed Findings by Grade and Subject. (Contains 49 tables.) [Supplemental support for the production of this report was provided by the Judith Gueron Fund for Methodological Innovation in Social Policy Research at MDRC. Dissemination of MDRC publications is also supported by The Starr Foundation.]
Descriptors: Program Effectiveness, Reading Achievement, Mathematics Achievement, Research Methodology, Urban Schools, Educational Assessment, Program Evaluation, Statistical Analysis, Pretests Posttests, Research Design
MDRC, 16 East 34th Street, 19th Floor, New York, NY 10016-4326. Tel: 212-532-3200; Fax: 212-684-0832; e-mail: email@example.com.
Publication Type: Information Analyses; Numerical/Quantitative Data; Reports - Research
Education Level: Grade 10; Grade 3; Grade 5; Grade 8
Audience: Practitioners; Policymakers
Sponsor: William T. Grant Foundation, New York, NY.; Alcoa Foundation, Pittsburgh, PA.; Ambrose Monell Foundation, New York, NY.; Atlantic Philanthropies; Bristol-Myers Squibb Foundation, Inc., New York, NY.; Open Society Inst., New York, NY.
Authoring Institution: Manpower Demonstration Research Corp., New York, NY.