NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED563359
Record Type: Non-Journal
Publication Date: 2013
Pages: 8
Abstractor: ERIC
Reference Count: 10
How Generalizable Is Your Experiment? An Index for Comparing Samples and Populations
Tipton, Elizabeth
Society for Research on Educational Effectiveness
Recent research on the design of social experiments has highlighted the effects of different design choices on research findings. Since experiments rarely collect their samples using random selection, in order to address these external validity problems and design choices, recent research has focused on two areas. The first area is on methods for adjusting experimental treatment effect estimates for known population characteristics. A second research area has been on how to select a sample (non-randomly) to improve generalizations to a well-defined inference population. The goal of this paper is to develop a measure that summarizes differences in the propensity score distributions in the experimental sample and population. The author includes an example, a middle school mathematics program SimCalc. SimCalc is a computer based program that teaches linearity, proportionality, and rates of change. The cluster randomized trial of this program was conducted on 73 schools in Texas. The primary purpose of this paper was to develop an index that can be used to evaluate how generalizable an experimental sample is for a particular inference population. This index is based entirely on pre-treatment variables, enabling the index both to be calculated and recalculated throughout recruitment but before an experiment has commenced (as a sampling tool) and to be calculated after an experiment has been completed for many different inference populations and outcomes. Figures are appended.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: Middle Schools; Secondary Education; Junior High Schools; Grade 7; Elementary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Texas