NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED563208
Record Type: Non-Journal
Publication Date: 2013
Pages: 6
Abstractor: ERIC
Reference Count: 10
Estimating Cross-Site Impact Variation in the Presence of Heteroscedasticity
Bloom, Howard S.; Porter, Kristin E.; Weiss, Michael J.; Raudenbush, Stephen
Society for Research on Educational Effectiveness
To date, evaluation research and policy analysis have focused mainly on average program impacts and paid little systematic attention to their variation. Recently, the growing number of multi-site randomized trials that are being planned and conducted make it increasingly feasible to study "cross-site" variation in impacts. Important technical questions arise when one uses data from a multi-site randomized trial to estimate the magnitude of cross-site impact variation and assess the statistical significance of these estimates. Some of these questions arise from the likelihood that variation in "individual" outcomes (which in the context of multi-level models is often referred to as a "level-one residual variance" or "residual variance" for short) might be different for the treatment group and control group members and/or might be different for different sites. In addition, to the extent that different populations and local conditions are represented by different sites, the variance of individual "counterfactual" outcomes (i.e. potential outcomes under no treatment) and therefore the variance of control group members' outcomes will differ across sites. This situation can be exacerbated if "the extent of" individual variation in program impacts also varies across sites because of their differences client populations and local conditions. These "differences in individual outcome variation" represent forms of heteroscedasticity which the authors refer to as: (1) T/C heteroscedasticity; and (2) cross-site heteroscedasticity. In this study, the following methodological questions are addressed through a series of simulations: To what extent are estimators of cross-site impact variation biased and tests of statistical significance incorrect when one does not take into account how the variance of individual outcomes differs between treatment and control group members and/or across sites? In other words, in practice, what are the risks of "estimating too few" residual variances and how does this risk vary under a wide range of conditions? (2) Among different estimators that take heteroscedasticity into account--estimators based on either a random-effects model, on a mixed-effects model or on no model (a method of moments estimator)--how do they compare? and (3) What recommendations can be made for practice?
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)