ERIC Number: EJ1125383
Record Type: Journal
Publication Date: 2017
Pages: 30
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1934-5747
EISSN: N/A
Empirical Performance of Covariates in Education Observational Studies
Wong, Vivian C.; Valentine, Jeffrey C.; Miller-Bains, Kate
Journal of Research on Educational Effectiveness, v10 n1 p207-236 2017
This article summarizes results from 12 empirical evaluations of observational methods in education contexts. We look at the performance of three common covariate-types in observational studies where the outcome is a standardized reading or math test. They are: pretest measures, local geographic matching, and rich covariate sets with a strong theory of treatment selection. Overall, the review demonstrates that although the pretest often reduces bias in observational studies, it does not always eliminate it. Its performance depends on the pretest's correlation with treatment selection and the outcome, and whether preintervention trends are present. We also find that although local comparisons are prioritized for matching, its performance depends on whether comparable no-treatment cases are available. Otherwise, local comparisons may produce badly biased results. In cases where researchers have a strong theory of selection and rich covariate sets, observational methods perform well, but additional replication studies are needed. Finally, observational methods that rely on demographic covariates without a theory of selection rarely produce unbiased treatment effects. The article concludes by offering education researchers empirically based guidance on covariate selection in observational studies.
Descriptors: Observation, Educational Research, Standardized Tests, Reading Tests, Mathematics Tests, Pretesting, Test Bias, Correlation, Intervention, Comparative Analysis, Research Methodology, Geographic Location, Literature Reviews, Sampling, Sample Size
Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research; Information Analyses
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: EDIES12C0084