NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED567592
Record Type: Non-Journal
Publication Date: 2016
Pages: 8
Abstractor: ERIC
Reference Count: 10
Analyzing Empirical Evaluations of Non-Experimental Methods in Field Settings
Steiner, Peter M.; Wong, Vivian
Society for Research on Educational Effectiveness
Despite recent emphasis on the use of randomized control trials (RCTs) for evaluating education interventions, in most areas of education research, observational methods remain the dominant approach for assessing program effects. Over the last three decades, the within-study comparison (WSC) design has emerged as a method for evaluating the performance of non-experimental methods in field settings. In a traditional WSC design, treatment effects from a randomized experiment (RE) are compared to those produced by a non-experimental approach that shares the same target population and intervention. The purpose of this paper is two-fold. First, it is to examine methods for assessing correspondence in experimental and non-experimental results in WSC designs, and their relative advantages and limitations. Second, it is to provide methods for calculating statistical power in WSC designs that evaluate the empirical performance of non-experimental methods. The authors apply multiple correspondence criteria to two WSC examples, highlighting contexts and conditions under which these methods converge at the same result, but also when they differ. Finally, the authors address correspondence criteria for WSC designs with independent experimental and nonexperimental conditions, and for cases when the conditions have dependent data structures. 2 figures are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)