ERIC Number: ED517991
Record Type: Non-Journal
Publication Date: 2011
Pages: 12
Abstractor: ERIC
ISBN: N/A
ISSN: N/A
EISSN: N/A
Extensions of Existing Methods for Use with a New Class of Experimental Designs Useful when There Is Treatment Effect Contamination
Rhoads, Christopher
Society for Research on Educational Effectiveness
Researchers planning a randomized field trial to evaluate the effectiveness of an educational intervention often face the following dilemma. They plan to recruit schools to participate in their study. The question is, "Should the researchers randomly assign individuals (either students or teachers, depending on the intervention) within schools to treatment conditions, or should all participating students at a given school be assigned to the same treatment condition?" That is, should they randomize schools (clusters), or individuals within schools? One reason often given for preferring cluster level randomization is a fear of "diffusion of treatment" (Raudenbush, 1997) or "contamination" (Donner and Klar, 2000). Contamination occurs when contact between members of the control group and members of the experimental group causes control group participants to behave more like experimental group participants than they would have had that contact not occurred. It is also possible for certain interventions that contamination could cause experimental subjects to behave more like control group subjects than they would otherwise. Note that both experimental subjects acting more like control subjects and control subjects acting like experimental subjects are processes that would tend to decrease the effect size of the experiment. The author assumes for the purposes of this paper that the contamination dilutes the observed effect size in this fashion. Cornfield (1978) noted that two penalties are paid for randomization by cluster rather than by individual. First, the variance of the estimated treatment effect increases. Second, the degrees of freedom available to estimate that variance decrease. Thus, in the absence of contamination, randomizing an equal number of individuals within each cluster to each treatment (often called a "randomized block" or RB design) is a more powerful design than randomizing whole clusters (often called a "cluster randomized" or CR design). Rhoads (forthcoming) has argued that the threat of contamination should not necessarily lead experimenters to opt for a cluster randomized design. He points out that, depending on the values of relevant design parameters (i.e., the ICC, within cluster sample size, the heterogeneity in treatment effects across clusters, and the number of clusters in the experiment) the statistical power of a randomized block design remains higher than the power of a cluster randomized design even when contamination causes the effect size to decrease by as much as 10-60%. Similarly, from the standpoint of mean squared error, Rhoads (forthcoming) shows that for many design parameters of practical interest the randomized block design will be preferred to the cluster randomized design. However, it may well be the case that the optimal experimental design in the presence of contamination is neither the RB design nor the CR design, but some compromise between the two designs. This is precisely the situation considered by Borm, Melis, Teerenstra and Peer (2005) (hereafter BMTP) who suggest an interesting compromise between cluster randomization and individual randomization, a method they call "pseudo cluster randomization." This research focuses on extending results found in BMTP (2005) to the following cases: (1) the case where the value of the ICC cannot be estimated precisely, (2) the case where treatment effects vary across clusters, (3) the case where the criterion function for determining an optimal design/estimator is mean squared error, rather than statistical power. The current study shows how the results given in the paper by BMTP can be generalized to the case where there are heterogeneous treatment effects. It further shows that when comparing cluster randomized and randomized block designs in the presence of contamination, one must pay careful attention to whether statistical power is the criterion for evaluating the design or if mean squared error is the criterion. Finally, it is shown that in pseudo-cluster randomized designs it is possible to create an estimator of the treatment effect whose variance does not depend on the unknown value of the ICC. One such design is the usual randomized block design. It is shown that one would choose to allocate an unequal number of individuals to treatment and control within each cluster only if the average contamination is expected to decrease substantially as a result. The further away from 1:1 allocation one goes, the more contamination must be reduced in order to prefer the unequal allocation. (Contains 3 tables.)
Descriptors: Research Design, Intervention, Effect Size, Outcomes of Treatment, Statistical Analysis, Experimental Groups, Control Groups, Scientific Methodology
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A