NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED562702
Record Type: Non-Journal
Publication Date: 2014
Pages: 12
Abstractor: ERIC
Reference Count: 5
Distortions in Distributions of Impact Estimates in Multi-Site Trials: The Central Limit Theorem Is Not Your Friend
May, Henry
Society for Research on Educational Effectiveness
Interest in variation in program impacts--How big is it? What might explain it?--has inspired recent work on the analysis of data from multi-site experiments. One critical aspect of this problem involves the use of random or fixed effect estimates to visualize the distribution of impact estimates across a sample of sites. Unfortunately, unless the sample sizes within sites are enormous (e.g., >1,000), the shape of the distribution of estimates is dominated by the normal distribution as a result of imperfect reliability in site-specific estimates. The Central Limit Theorem explains why this occurs and why the problem is unavoidable. The purpose of this paper is to demonstrate the scope and severity of this problem, and to provide cautionary guidance regarding the interpretation of distributions of site-specific impact estimates in multi-site experiments. Furthermore, the paper explores methods for recovering the distribution of true site-specific impacts when that distribution is non-normal. For illustrative purposes, data from the multi-site randomized trial under the Reading Recovery Investing in Innovation (i3) Scale-Up is used in addition to generalized Monte Carlo simulation. Reading Recovery is an early reading intervention for first-graders who are reading substantially below grade level. Data figures are appended.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: Grade 1; Primary Education; Elementary Education; Early Childhood Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)