The Cooperative Extension Service has been a key partner in the design, implementation and evaluation of leadership development programs. To evaluate the effectiveness of their training and the effects of response shift bias on outcomes using a self-report measure, one hundred forty-seven County Extension Agents participated in this leadership study. Participants were randomly assigned to one of two treatment groups or to a control group. Two different evaluation designs (pre-posttest and then-post) were used. The then-post design asks participants to first report their behavior or understanding as a result of the training (post) and then to retrospectively report this behavior before the training. The then-post evaluation design provided more significant change data than did the traditional pre-posttest design indicating a response shift occurred. Such differences in evaluation findings suggest that the educational benefit of such trainings may be underestimated when using the traditional pre-post evaluation design.