This study aimed to assess the impact of an entire academic teacher development programme at a Midwestern masters comprehensive university in the United States over a period of five years by examining changes in teaching and student outcomes of nine randomly selected programme participants. Researchers analysed syllabi, course evaluations, grade distributions, programme activity reports, interviews and vita, and concluded that the centre's impact was greatest on tenure-track faculty who struggled with their teaching. Far more useful to future improvement, however, were the insights gained from reflection on study methods, the researchers' emotional responses to their findings, their struggle to make sense of the data and grappling with the definition of 'impact'. In this paper, the researchers share useful lessons learned and illustrate the difficulties inherent in evaluating academic development programmes, with a view to qualifying recommendations currently prevailing in the literature.