We evaluated the effectiveness of Boston vs. Bullies, a short-term, free, bullying prevention program that uses celebrity athletes to present content. Fifth-grade students in 10 schools were randomized to either complete the Boston vs. Bullies intervention (n = 388), or to a wait-list control group (n = 266). Pre- and post-surveys assessed knowledge, attitudes, and behaviors related to bullying. Students completing Boston vs. Bullies reported greater improvement in knowledge of bullying, assertiveness, perceptions of adult responsiveness, and bystander responsibility. They also reported decreased acceptance of aggression and peer victimization. However, when statistical models introduced robust standard errors to account for school clustering, some associations attenuated, suggesting that program effectiveness is somewhat variable across schools. Further, among youth in the intervention group, greater improvement was associated with student-reported engagement and facilitator-reported adherence to program components. Results suggest that Boston vs. Bullies can contribute to improving bullying, but some program outcomes may be influenced by school context.