NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED564087
Record Type: Non-Journal
Publication Date: 2013
Pages: 8
Abstractor: ERIC
Reference Count: 14
Prognostic Score-Based Difference-in-Differences Strategy for Multilevel Multi-Cohort Data
Hong, Guanglei
Society for Research on Educational Effectiveness
When using time series accountability data to evaluate system-wide education policies, concurrent changes often pose threats to internal validity. The standard difference-in-differences (DID) method resorts to a non-equivalent comparison group whose average outcome change is due to such confounding. This strategy relies on the strong assumption that the average confounding impact of concurrent events is the same for the comparison group unaffected by the policy and the experimental group affected by the policy. This assumption will be violated and therefore the DID results will be biased, for example, if the confounding effect varies by individual characteristics and if the experimental group and the comparison group differ in such characteristics. Prior research has typically employed a DID model with linear covariance adjustment for observed pretreatment characteristics. More recently, researchers have attempted to equate the covariate distribution of the comparison group with that of the experimental group through propensity score matching or weighting before conducting DID analyses. This study provides the theoretical rationale for the prognostic score-based DID strategy, clarifies its identification assumptions, and develops an analytic procedure. This study illustrates the new strategy with an evaluation of a policy adopted by the Chicago Public Schools requiring all ninth graders to take algebra, defines the causal estimand, and develops statistical models for investigating whether the policy effect was enhanced as its implementation became mature or whether the effect faded out over time as the reform lost its momentum after the initial period. The new strategy is then extended to multilevel multi-cohort education accountability data. This new strategy greatly reduces the dimensionality of covariates for adjustment, which is a major advantage over the linear covariance adjusted DID. The stratification procedure enables researchers to detect heterogeneity in the confounding effects of concurrent events as well as in the policy effect. Yet like most other DID methods, this new strategy shows limitations when the experimental group and the conditional group differ in the distribution of an unobservable and when the amount of confounding of concurrent events is a function of the unobservable. Appendices include a proof and information on obtaining an estimate of the policy effect on the untreated pre-policy students in the experimental school.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail:; Web site:
Publication Type: Reports - Research
Education Level: Grade 9; Junior High Schools; Middle Schools; Secondary Education; High Schools
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Illinois