Education has become one of the most important issues in American society, often dominating the political landscape in response to lagging academic achievement and persistent gaps in performance between advantaged and disadvantaged children. In response, a variety of educational reforms have been implemented ranging from increased accountability to parental school choice and efforts to improve educational "inputs" such as teacher skills. An alternative strategy that has gained in popularity involves efforts to expand public and private investments in out-of-school-time programs, particularly those targeting the needs of disadvantaged children. Such programs provide opportunities for children and teens to have positive interactions with caring adults, improve socialization with other children, learn to avoid risky behaviors, and, last but not least, improve academic skills. However, students spend the majority of their time in these programs participating in extracurricular activities that are not designed specifically to improve academic skills. Consequently, this study asks two important research questions: (1) Does participation in out-of-school extracurricular activities improve academic achievement or behavior for elementary school children? and (2) If so, are the impacts of participation related to the types of extracurricular activities that students pursue (e.g., music and arts, language, computer classes, sports)? Initial analyses were conducted using relatively standard techniques to control for the fact that students who participate in out-of-school-time extracurricular activities differ in important ways from nonparticipants. These analyses found statistically significant and positive effects of such participation in arts, music, drama, and language classes. We were ready to write our report at this stage, but then tried to replicate our analyses using alternative statistical models. The results were not only highly dependent on the choice of analytical approach, but the initial positive effects disappeared when more appropriate analytical models were used. We then examined the current literature that has been used to support investments in out-of-school-time extracurricular programs and found that the reported positive effects are likely a result of the same type of analytical error that led us to initially conclude that there were positive effects for elementary school children. At a minimum, these results raise serious questions about the validity of many of the claims that out-of-school-time programs that do not directly target academic outcomes will improve such outcomes nonetheless. The results also suggest that more rigorous evaluation be conducted before further investments are made based on such claims, particularly where such expenditures are made as a trade-off against investments in other, possibly more effective, educational reforms. Four appendixes are included: (1) Detailed Descriptive Statistics; (2) Behavioral Scales Used in the Analysis; (3) Measurement Error and IV Models; and (4) Defining After-School, Out-of-School, and Enrichment. (Contains 54 footnotes, 1 figure, and 11 tables.)