ERIC Number: EJ1175213
Record Type: Journal
Publication Date: 2018
Pages: 29
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1934-5747
EISSN: N/A
Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers
Porter, Kristin E.
Journal of Research on Educational Effectiveness, v11 n2 p267-295 2018
Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs) are statistical procedures that counteract this problem by adjusting "p" values for effect estimates upward. Although MTPs are increasingly used in impact evaluations in education and other areas, an important consequence of their use is a change in statistical power that can be substantial. Unfortunately, researchers frequently ignore the power implications of MTPs when designing studies. Consequently, in some cases, sample sizes may be too small, and studies may be underpowered to detect effects as small as a desired size. In other cases, sample sizes may be larger than needed, or studies may be powered to detect smaller effects than anticipated. This paper presents methods for estimating statistical power for multiple definitions of statistical power and presents empirical findings on how power is affected by the use of MTPs. [For the full-text grantee submission, see ED567783.]
Descriptors: Statistical Analysis, Program Effectiveness, Intervention, Hypothesis Testing, Evaluation Methods, Effect Size, Sample Size, Computation, Correlation, Educational Research, Monte Carlo Methods
Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Guides - Non-Classroom
Education Level: N/A
Audience: Researchers
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R305D140024