Beginning in 2002, researchers developed, implemented, and evaluated the efficacy of an English reading intervention for first-grade English learners using multiple randomized control trials (RCTs). As a result of this efficacy work, researchers successfully competed for an IES Goal 4 effectiveness study using the same intervention. Unlike the efficacy studies, the findings from the effectiveness study begun in 2011 did not yield significant differences in favor of the treatment. We investigated the identification and pre-intervention performance levels of students in efficacy and effectiveness RCT studies and provide data and compare samples from these studies as one means of explaining variation in findings. Findings indicate that pre-intervention performance levels are higher for students in effectiveness RCT studies. This article serves as an exemplar for other effectiveness studies that may not replicate findings from previous efficacy trials.