Since 2001 the National Science Foundation's ADVANCE program has distributed over $130 million in grants to improve work climate, enhance professional success, and increase recruitment and retention of female faculty in STEM fields. The process by which each institution designs and implements these interventions is seldom studied, however. Using climate surveys, administrative records, and a difference-in-differences regression approach, we assessed whether exposure to the design and implementation process helps explain improvements in climate and retention during the early years of ADVANCE implementation. We found that departments wherein at least one faculty member participated in ADVANCE committee work experienced significant improvements in job satisfaction among female faculty members and significant reduction in turnover among female full professors, suggesting that the ADVANCE design process was itself an intervention.