ERIC Number: ED462524
Record Type: Non-Journal
Publication Date: 2002-Jan
How Close Is Close Enough? Testing Nonexperimental Estimates of Impact against Experimental Estimates of Impact with Education Test Scores as Outcomes. Discussion Paper.
Wilde, Elizabeth Ty; Hollister, Robinson
This study tested the performance of nonexperimental estimators of impacts applied to a class size reduction intervention with achievement test scores as the outcome. Nonexperimental estimates of impacts were compared to "true impact" estimates provided by a random-assignment design that assessed intervention effects. Data came from Project STAR, a study in which K-3 students were randomly assigned to small or regular-size classes in 79 Tennessee schools. Outcome variables were math and reading achievement scores. The propensity-score-matching estimating procedure separately for each of 11 schools' kindergartens was conducted, then nonexperimental estimates of the impact of smaller class size were derived. Proper standard errors for the propensity-score-matched estimators using bootstrapping procedures were then developed. In general, the propensity-score estimate of the impact differed substantially from the "true impact" initially estimated. How close nonexperimental estimates were to experimental estimates was assessed, suggesting several different ways of assessing "closeness." Most ways led to the conclusion that the nonexperimental estimates were not very "close" and therefore were not reliable guides to the "true impact." In 30-45 percent of the 11 cases, the propensity-score-matching nonexperimental estimators would have led to the "wrong" decision. (Contains 10 references and 5 tables.) (SM)
Descriptors: Academic Achievement, Class Size, Educational Research, Elementary Secondary Education, Research Methodology
Publications Department, Institute for Research on Poverty, 1180 Observatory Drive, Madison, WI 53706-1393 ($3.50). Tel: 608-262-6358; Fax: 608-265-3119; e-mail: email@example.com; Web site: http://www.ssc.wisc.edu/irp. For full text: http://www.ssc.wisc.edu/irp/pubs/dp124202.pdf.
Publication Type: Reports - Descriptive; Reports - Research
Education Level: N/A
Authoring Institution: Wisconsin Univ., Madison. Inst. for Research on Poverty.