In this article, we test an alternative approach to creating a counterfactual basis for estimating individual and average treatment effects. Instead of using control/comparison groups or before-measures, the so-called Counterfactual as Self-Estimated by Program Participants (CSEPP) relies on program participants' self-estimations of their own…

This classification study examines evaluators' coursework training as a way of understanding evaluation practice. Data regarding courses that span methods and evaluation topics were collected from evaluation practitioners. Using latent class analysis, we establish four distinct classes of evaluator course-taking patterns: quantitative,…

In this paper we share our reflections, as evaluators, on an evaluation where we encountered Excessive Evaluation Anxiety (XEA). The signs of XEA which we discerned were particularly evident amongst the program head and staff who were part of a new training program. We present our insights on the evaluation process and its difficulties, as well as…

Extracted from a larger study of the educational evaluation profession, this qualitative analysis explores how evaluator identity is shaped with constant reference to political economy, knowledge work, and personal history. Interviews with 24 social scientists who conduct or have conducted evaluations as a major part of their careers examined how…

Quality training opportunities for evaluators will always be important to the evaluation profession. While studies have documented the number of university programs providing evaluation training, additional information is needed concerning what content is being taught in current evaluation courses. This article summarizes the findings of a survey…

This article proposes an innovative approach to estimating the counterfactual without the necessity of generating information from either a control group or a before-measure. Building on the idea that program participants are capable of estimating the hypothetical state they would be in had they not participated, the basics of the Roy-Rubin model…

In this Exemplars case, the fifth and final under the direction of the current coeditors, the authors present a reflective account of an ongoing, complex, multiyear, multinational monitoring and evaluation (M&E) system conducted for African Women in Agricultural Research and Development (AWARD), an international development program. The…

Program evaluation is recognized as an essential skill set for practitioners in service-related fields, such as education, nonprofit management, social work, and public health. Recently, the need for a public workforce trained in evaluation has increased and is driven primarily by our nation's emphasis on accountability during a time when…

Shadowing is a data collection method that involves following a person, as they carry out those everyday activities relevant to a research study. This article explores the use of shadowing in a formative evaluation of a professional development school (PDS). Specifically, this article discusses how shadowing was used to understand the role of a…

This article explores the viability of online crowdsourcing for creating matched-comparison groups. This exploratory study compares survey results from a randomized control group to survey results from a matched-comparison group created from Amazon.com's MTurk crowdsourcing service to determine their comparability. Study findings indicate…

In this commentary, we examine Braverman's insights into the trade-offs between feasibility and rigor in evaluation measures and reject his assessment of the trade-off as a zero-sum game. We, argue that feasibility and policy salience are, like reliability and validity, intrinsic to the definition of a good measure. To reduce the tension between…

Joseph S. Wholey is Professor Emeritus in the University of Southern California's (USC) School of Public Policy, Planning, and Development. He received his BA from Catholic University, Phi Beta Kappa and his MA in mathematics and PhD in philosophy from Harvard University. His work focuses on the use of strategic planning, performance measurement,…

Evaluators' experiences with being pressured by a key stakeholder to misrepresent findings were investigated in an anonymous survey of a random sample of over 2500 American Evaluation Association members; a response rate of 37% was obtained. Overall, 42% of the respondents had encountered misrepresentation pressure, with 70% of this subgroup…

This article presents the results of a survey conducted in an attempt to examine evaluators' perceptions of the contributions that different educational experiences make to the development of their evaluation skill sets. Self-identified new and graduate student evaluators were asked to describe their evaluation training experiences and to indicate…

Evaluator training field experiences seek to reinforce applicable content knowledge and promote applied skills. For students, even a small-scale field experience focusing on limited components of a larger evaluation process can seem particularly challenging. Students often do not recognize the relevance of established evaluation resources capable…