ERIC Number: EJ1141300
Record Type: Journal
Publication Date: 2017-Jun
Abstractor: As Provided
Variability in the Results of Meta-Analysis as a Function of Comparing Effect Sizes Based on Scores from Noncomparable Measures: A Simulation Study
Nugent, William R.
Educational and Psychological Measurement, v77 n3 p449-470 Jun 2017
Meta-analysis is a significant methodological advance that is increasingly important in research synthesis. Fundamental to meta-analysis is the presumption that effect sizes, such as the standardized mean difference (SMD), based on scores from different measures are comparable. It has been argued that population observed score SMDs based on scores from different measures A and B will be equal only if the conjunction of three conditions are met: construct equivalence (CE), equal reliabilities (ER), and the absence of differential test functioning (DTF) in all subpopulations of the combined populations of interest. It has also been speculated the results of a meta-analysis of SMDs might differ between circumstances in which the SMDs included in a meta-analysis are based on measures which all met the conjunction of these conditions and that in which the conjunction of these conditions is violated. No previous studies have tested this conjecture. This Monte Carlo study investigated this hypothesis. A population of studies comparing one of five hypothetical treatments with a placebo condition was simulated. The SMDs in these simulated studies were based on true scores from six hypothetical measures. The scores from some of these measures met the conjunction of CE, ER, and, the absence of DTF, while others failed to meet CE. Three meta-analyses were conducted using both fixed effects and random effects methods. The results suggested that the results of meta-analyses can vary to a practically significant degree when the SMDs were based on scores from measures failing to meet the CE condition. Implications for future research are considered.
Descriptors: Meta Analysis, Effect Size, Comparative Analysis, Scores, Simulation, Monte Carlo Methods
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: email@example.com; Web site: http://sagepub.com
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A