This research evaluated the impact of a common modification to Angoff standard-setting exercises: the provision of examinee performance data. Data from 18 independent standard-setting panels across three different medical licensing examinations were examined to investigate whether and how the provision of performance information impacted judgments…

Evidence of stable standard setting results over panels or occasions is an important part of the validity argument for an established cut score. Unfortunately, due to the high cost of convening multiple panels of content experts, standards often are based on the recommendation from a single panel of judges. This approach implicitly assumes that…

The purpose of the present study was to extend past work with the Angoff method for setting standards by examining judgments at the judge level rather than the panel level. The focus was on investigating the relationship between observed Angoff standard setting judgments and empirical conditional probabilities. This relationship has been used as a…

One of the best-known methods for setting a benchmark standard on a test is that of Angoff and its modifications. When scored dichotomously, judges estimate the probability that a benchmark student has of answering each item correctly. As in most methods of standard setting, it is assumed implicitly that the unit of the latent scale of the…

This article describes a novel variation of the Body of Work method that uses construct maps to overcome problems of transparency, rater inconsistency, and scores gaps commonly occurring with the Body of Work method. The Body of Work method with construct maps was implemented to set cut-scores for two separate K-12 assessment programs in a large…

Construct maps are tools that display how the underlying achievement construct upon which one is trying to set cut-scores is related to other information used in the process of standard setting. This article reviews what construct maps are, uses construct maps to provide a conceptual framework to view commonly used standard-setting procedures (the…

The authors of this article comment on "Construct Maps as a Foundation for Standard Setting," by Adam E. Wyse (this issue) in which Wyse presents construct maps, a visual display of a variety of sources of evidence that support standard-setting decisions, and shows how this approach could be used with a variety of existing…

The construct map is a promising tool for organizing the data standard-setting panelists interpret. The challenge in applying construct maps to standard-setting procedures will be the judicious selection of data to include within this organizing framework. Therefore, this commentary focuses on decisions about what to include in the construct map.…

The author recalls that as a child, he grooved on teeter-totters. Also known as a seesaw, a teeter-totter is a long, narrow board that's elevated with a pivot point in the middle so that as one end goes down the other end goes up. When going up or going down, sometimes quite rapidly, teeter-totters can provide their two riders with some…

The construct map is a particularly good way to approach instrument development, and this author states that he was delighted to read Adam Wyse's thoughts about how to use construct maps for standard setting. For a number of popular standard-setting methods, Wyse shows how typical feedback to panelists fits within a construct map framework.…

The authors observe in this commentary that construct maps can help standard-setting panels to make realistic and internally consistent recommendations for performance-level descriptions (PLDs) and cut-scores, but the benefits may not be realized if policymakers do not fully understand the rationale for the recommendations provided by the…

In this issue of "Measurement: Interdisciplinary Research and Perspectives," Adam E. Wyse provides a thorough review of research to date on the use of construct maps in standard setting. He juxtaposes concepts and methods in ways that make their connections to one another clearer and more obvious than they might otherwise have been. In…

In his article "Construct Maps for Standard Setting," Adam E. Wyse provides a detailed review on the current use of construct maps in standard setting, including how they may be operationalized within a variety of standard-setting methods. The premise of the argument is that construct maps can serve as a useful tool for conducting a…

In this article, E. Matthew Schulz responds to Adam Wyse's article, "Construct Maps as a Foundation for Standard Setting." In doing so, he asserts that one of the most important ideas in Wyse's work is that information used in standard setting needs to be better represented through the use of graphics. However, he's not…

Despite being widely used and frequently studied, the Angoff standard setting procedure has received little attention with respect to an integral part of the process: how judges incorporate examinee performance data in the decision-making process. Without performance data, subject matter experts have considerable difficulty accurately making the…