NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1100453
Record Type: Journal
Publication Date: 2016-Jun
Pages: 18
Abstractor: As Provided
ISSN: ISSN-0162-3737
Estimates of External Validity Bias When Impact Evaluations Select Sites Nonrandomly
Bell, Stephen H.; Olsen, Robert B.; Orr, Larry L.; Stuart, Elizabeth A.
Educational Evaluation and Policy Analysis, v38 n2 p318-335 Jun 2016
Evaluations of educational programs or interventions are typically conducted in nonrandomly selected samples of schools or districts. Recent research has shown that nonrandom site selection can yield biased impact estimates. To estimate the external validity bias from nonrandom site selection, we combine lists of school districts that were selected nonrandomly for 11 educational impact studies with population data on student outcomes from the Reading First program. Our analysis finds that on average, if an impact study of Reading First were conducted in the districts from these 11 studies, the impact estimate would be biased downward. In particular, it would be 0.10 standard deviations lower than the impact in the broader population from which the samples were selected, a substantial bias based on several benchmarks of comparison.
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED); National Science Foundation (NSF); National Institute of Mental Health (DHHS/NIH)
Authoring Institution: N/A
Identifiers - Location: Arizona; Colorado; Delaware; Florida; Georgia; Hawaii; North Carolina; South Carolina; Texas
IES Funded: Yes
Grant or Contract Numbers: R305D100041; DRL1335843; K25MH083846