ERIC Number: ED630841
Record Type: Non-Journal
Publication Date: 2019
Pages: 28
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Precise Unbiased Estimation in Randomized Experiments Using Auxiliary Observational Data
Gagnon-Bartsch, J. A.; Sales, A. C.; Wu, E.; Botelho, A. F.; Erickson, J. A.; Miratrix, L. W.; Heffernan, N. T.
Grantee Submission
Randomized controlled trials (RCTs) admit unconfounded design-based inference--randomization largely justifies the assumptions underlying statistical effect estimates--but often have limited sample sizes. However, researchers may have access to big observational data on covariates and outcomes from RCT non-participants. For example, data from A/B tests conducted within an educational technology platform exist alongside historical observational data drawn from student logs. We outline a design-based approach to using such observational data for variance reduction in RCTs. First, we use the observational data to train a machine learning algorithm predicting potential outcomes using covariates, and use that algorithm to generate predictions for RCT participants. Then, we use those predictions, perhaps alongside other covariates, to adjust causal effect estimates with a flexible, design-based covariate-adjustment routine. In this way there is no danger of biases from the observational data leaking into the experimental estimates, which are guaranteed to be exactly unbiased regardless of whether the machine learning models are "correct" in any sense or whether the observational samples closely resemble RCT samples. We demonstrate the method in analyzing 33 randomized A/B tests, and show that it decreases standard errors relative to other estimators, sometimes substantially. [This is the online version of an article published in "Journal of Causal Inference." Additional funding was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN) program.]
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED); National Science Foundation (NSF); Office of Elementary and Secondary Education (OESE) (ED), Education Innovation and Research (EIR); Office of Naval Research (ONR) (DOD); Department of Education (ED)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R305D210031; DMS1646108; 2118725; 2118904; 1950683; 1917808; 1931523; 1940236; 1917713; 1903304; 1822830; 1759229; 1724889; 1636782; 1535428; R305N210049; R305A170137; R305A170243; R305A180401; R305D210036; R305A120125; R305R220012; U411B190024; S411B210024; N000141812768; R305A170641
Data File: URL: https://osf.io/d9ujq/
Author Affiliations: N/A