ERIC Number: ED452237
Record Type: Non-Journal
Publication Date: 2000-Nov-17
Reference Count: N/A
Avoiding Decision-Making by Chance: Protecting Effect Size Estimates.
Barnette, J. Jackson; McLean, James E.
The probabilities of attaining varying magnitudes of standardized effect sizes by chance and when protected by a 0.05 level statistical test were studied. Monte Carlo procedures were used to generate standardized effect sizes in a one-way analysis of variance situation with 2 through 5, 6, 8, and 10 groups with selected sample sizes from 5 to 500. Within each of the 91 group and sample size configurations, 100,000 replications were generated from a distribution of normal deviates. For each data set, the effect size was computed along with a statistical test of the hypothesis at the 0.05 level. For each n/k combination, the proportion of effect sizes exceeding 0.1 to 2.0 in increments of 0.1 was computed for all cases and for those cases where "the no difference hypothesis" was rejected. There were trends that were common across all configurations. As the magnitude of effect size increased, the probability of getting such a difference by chance decreased, as would be expected. Within a given number of samples situation, as sample size increased, as expected, the probability of getting such a difference by chance decreased. Within a given sample size, as the number of groups increased, the probability of getting such a difference by chance increased. Another finding that was consistent across all configurations was that the significance test protected effect size probability was always equal to or less than the unprotected probability, in some cases dramatically so. It was clear that the addition of the significance test reduced the probability of finding a seemingly large effect size by chance. Such a protected effect size indicator could be an answer to the arguments posed by both those who protest against the use of the significance test and those who propose its use in judging the magnitude of an observed effect. (Contains 15 tables, 14 figures, and 43 references.) (Author/SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A