ERIC Number: EJ885768
Record Type: Journal
Publication Date: 2010-Jun
Abstractor: As Provided
Reference Count: 70
A Model-Averaging Approach to Replication : The Case of "p[subscript rep]"
Iverson, Geoffrey J.; Wagenmakers, Eric-Jan; Lee, Michael D.
Psychological Methods, v15 n2 p172-181 Jun 2010
The purpose of the recently proposed "p[subscript rep]" statistic is to estimate the probability of concurrence, that is, the probability that a replicate experiment yields an effect of the same sign (Killeen, 2005a). The influential journal "Psychological Science" endorses "p[subscript rep]" and recommends its use over that of traditional methods. Here we show that "p[subscript rep]" overestimates the probability of concurrence. This is because "p[subscript rep]" was derived under the assumption that all effect sizes in the population are equally likely a priori. In many situations, however, it is advisable also to entertain a null hypothesis of no or approximately no effect. We show how the posterior probability of the null hypothesis is sensitive to a priori considerations and to the evidence provided by the data; and the higher the posterior probability of the null hypothesis, the smaller the probability of concurrence. When the null hypothesis and the alternative hypothesis are equally likely a priori, "p[subscript rep]" may overestimate the probability of concurrence by 30% and more. We conclude that "p[subscript rep]" provides an upper bound on the probability of concurrence, a bound that brings with it the danger of having researchers believe that their experimental effects are much more reliable than they actually are. (Contains 4 footnotes and 2 figures.)
Descriptors: Effect Size, Evaluation Methods, Probability, Experiments, Reliability, Experimental Psychology, Replication (Evaluation), Evaluation Research, Evaluation Problems, Misconceptions, Bayesian Statistics, Hypothesis Testing, Predictive Measurement, Measurement Techniques, Models, Statistical Inference, Priming
American Psychological Association. Journals Department, 750 First Street NE, Washington, DC 20002-4242. Tel: 800-374-2721; Tel: 202-336-5510; Fax: 202-336-5502; e-mail: firstname.lastname@example.org; Web site: http://www.apa.org/publications
Publication Type: Journal Articles; Reports - Descriptive
Education Level: N/A
Authoring Institution: N/A