ERIC Number: ED502199
Record Type: Non-Journal
Publication Date: 2008-May
Guidelines for Multiple Testing in Impact Evaluations of Educational Interventions. Final Report
Schochet, Peter Z.
Mathematica Policy Research, Inc.
Studies that examine the impacts of education interventions on key student, teacher, and school outcomes typically collect data on large samples and on many outcomes. In analyzing these data, researchers typically conduct multiple hypothesis tests to address key impact evaluation questions. Tests are conducted to assess intervention effects for multiple outcomes, for multiple subgroups of schools or individuals, and sometimes across multiple treatment alternatives. Multiple comparisons issues are not frequently addressed in impact evaluations of educational interventions. The Institute of Education Sciences (IES) at the U.S. Department of Education (ED) contracted with Mathematica Policy Research, Inc. (MPR) to develop guidelines for appropriately handling multiple testing in education research. This report provides a structure to address the multiplicity problem and discusses issues to consider when formulating a testing strategy. Four appendixes are included: (1) Panel Members; (2) Introduction to Multiple Testing; (3) Weighting Options for Constructing Composite Domain Outcomes; and (4) Bayesian Hypothesis Testing Framework (alternative to the classical hypothesis testing framework that is assumed for this report.) (Contains 5 footnotes and 5 tables.)
Descriptors: Hypothesis Testing, Guidelines, Outcomes of Education, Evaluation Methods, Program Evaluation, Educational Research, Research Methodology, Statistical Analysis, Experimental Groups, Control Groups, Error Patterns, Factor Analysis, Comparative Analysis
Mathematica Policy Research, Inc. P.O. Box 2393, Princeton, NJ 08543-2393. Tel: 609-799-3535; Fax: 609-799-0005; e-mail: firstname.lastname@example.org; Web site: http://www.mathematica-mpr.com/publications/
Publication Type: Reports - Descriptive
Education Level: N/A
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: Mathematica Policy Research, Inc.
Grant or Contract Numbers: N/A