NotesFAQContact Us
Collection
Advanced
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED562724
Record Type: Non-Journal
Publication Date: 2014
Pages: 14
Abstractor: ERIC
Reference Count: 18
ISBN: N/A
ISSN: N/A
Causal Inference and the Comparative Interrupted Time Series Design: Findings from Within-Study Comparisons
St. Clair, Travis; Hallberg, Kelly; Cook, Thomas D.
Society for Research on Educational Effectiveness
Researchers are increasingly using comparative interrupted time series (CITS) designs to estimate the effects of programs and policies when randomized controlled trials are not feasible. In a simple interrupted time series design, researchers compare the pre-treatment values of a treatment group time series to post-treatment values in order to assess the impact of a treatment, without any comparison group to account for confounding factors. The CITS design is a version of the ITS design in which both a treatment and a comparison group are evaluated both before and after the onset of a treatment. A growing body of literature is employing a within study comparison (WSC) methodology to examine the validity of the CITS model. WSC studies empirically estimate the extent to which a given observational study reproduces the results of a randomized controlled trial (RCT) when both share the same treatment group, and represent a rigorous method of evaluating non-experimental designs using real data. A number of recent within-study comparisons have demonstrated that CITS can produce estimates that are comparable to those from a randomized controlled trial (RCT) in practice. In the St. Clair et al. (2014) application, the authors found that correspondence with the RCT was possible when the CITS model accounted for baseline trends, but that additional time points could actually increase bias when the pre-treatment trend was not modeled correctly. Examination of the pretreatment trends in this data set showed clearly that in at least one of the outcomes the treatment and comparison groups had different slopes in the pretreatment period, and as a result the "parallel trends" assumption often invoked in the difference-in-difference literature was clearly violated. This paper employs a within study comparison (WSC) methodology to examine the performance of two approaches: (1) a more flexible modeling approach, which employs year fixed-effects rather than trying to parametrically model the pretest trend; and (2) match treatment and comparison cases to reduce reliance on modeling the pretreatment trend. The paper then compares the approaches to the performance of the baseline mean and baseline slope models across three datasets. The purpose of this research is two-fold: (1) to examine what approach, if any, works in the unclear functional form case; and (2) to examine the relative superiority of the different approaches across the three datasets in terms of both bias reduction and precision. Tables and figures are appended.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org
Publication Type: Reports - Research
Education Level: Grade 4; Intermediate Grades; Elementary Education; Grade 5; Middle Schools; Grade 6
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Florida
Identifiers - Assessments and Surveys: Indiana Statewide Testing for Educational Progress Plus