In light of the current debate over the meaning of "scientifically based research", we argue that qualitative methods should be an essential part of large-scale program evaluations if program effectiveness is to be determined and understood. This article chronicles the challenges involved in incorporating qualitative methods into the large-scale evaluation of the Waterford Early Reading Program and the decisions made in light of those challenges. We demonstrate that, in spite of the challenges, there are significant benefits associated with using qualitative methods on a large scale. More specifically, by using qualitative methods, we were able to improve the evaluation and generate findings that were meaningful and useful to stakeholders. It is our hope that our experiences can be used by others to inform their work as they seek to incorporate qualitative methods into their large-scale evaluations.