ERIC Number: EJ1162254
Record Type: Journal
Publication Date: 2017
Pages: 17
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0022-0655
EISSN: N/A
An Experimental Study of the Internal Consistency of Judgments Made in Bookmark Standard Setting
Clauser, Brian E.; Baldwin, Peter; Margolis, Melissa J.; Mee, Janet; Winward, Marcia
Journal of Educational Measurement, v54 n4 p481-497 Win 2017
Validating performance standards is challenging and complex. Because of the difficulties associated with collecting evidence related to external criteria, validity arguments rely heavily on evidence related to internal criteria--especially evidence that expert judgments are internally consistent. Given its importance, it is somewhat surprising that evidence of this kind has rarely been published in the context of the widely used bookmark standard-setting procedure. In this article we examined the effect of ordered item booklet difficulty on content experts' bookmark judgments. If panelists make internally consistent judgments, their resultant cut scores should be unaffected by the difficulty of their respective booklets. This internal consistency was not observed: the results suggest that substantial systematic differences in the resultant cut scores can arise when the difficulty of the ordered item booklets varies. These findings raise questions about the ability of content experts to make the judgments required by the bookmark procedure.
Descriptors: Evaluation Methods, Standard Setting, Cutting Scores, Expertise, Test Items, Test Validity, Methods Research
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A