ERIC Number: EJ1019034
Record Type: Journal
Publication Date: 2013-Dec
Pages: 22
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1098-2140
EISSN: N/A
Available Date: N/A
On Overfitting in Analysis of Symmetrically Predicted Endogenous Subgroups from Randomized Experimental Samples: Part Three of a Method Note in Three Parts
Harvill, Eleanor L.; Peck, Laura R.; Bell, Stephen H.
American Journal of Evaluation, v34 n4 p545-566 Dec 2013
Using exogenous characteristics to identify endogenous subgroups, the approach discussed in this method note creates symmetric subsets within treatment and control groups, allowing the analysis to take advantage of an experimental design. In order to maintain treatment--control symmetry, however, prior work has posited that it is necessary to use a prediction subsample, separate from the subsample used for impact estimation in order to prevent overfitting from affecting impact estimates. Doing so diminishes sample size--both for prediction and analysis--and so has costs. This article delves into this topic to consider the conditions under which overfitting occurs and to characterize the effects of overfitting in terms of bias and variance. It suggests a strategy for preserving the full sample size in all phases of the analysis. The research uses Monte Carlo simulation to directly measure overfitting, identify the circumstances that should concern us, and to explore possible recommended practices and future research implications. [To access part two of this method note entitled: "Using Symmetric Predication of Endogenous Subgroups for Causal Inferences about Program Effects under Robust Assumptions: Part Two of a Method Note in Three Parts," see EJ1015416.]
Descriptors: Experimental Groups, Control Groups, Research Design, Sampling, Sample Size, Monte Carlo Methods, Prediction, Causal Models, Inferences, Program Evaluation, Program Effectiveness, Simulation, Evaluation Methods, Policy, Intervention, Least Squares Statistics
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Publication Type: Reports - Research; Journal Articles
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A