ERIC Number: EJ1067854
Record Type: Journal
Publication Date: 2015-Aug
Abstractor: As Provided
Reference Count: 16
Much Ado about Differences: Why Expert-Novice Comparisons Add Little to the Validity Argument
Cook, David A.
Advances in Health Sciences Education, v20 n3 p829-834 Aug 2015
One approach to validating assessment scores involves evaluating the ability of scores to discriminate among groups who differ in a specific characteristic, such as training status (in education) or disease state (in clinical applications). Such known-groups comparison studies provide validity evidence of "relationships with other variables." The typical education research study might compare scores between staff physicians and postgraduate trainees with the hypothesis that those with more advanced training (the "experts") will have higher scores than those less advanced (the "novices"). However, such comparisons are too nonspecific to support clear conclusions, and expert-novice comparisons (and known-groups comparisons in general) thus contribute little to the validity argument. The major flaw is the problem of confounding: there are multiple plausible explanations for any observed between-group differences. The absence of hypothesized differences would suggest a serious flaw in the validity argument, but the confirmation of such differences adds little. As such, accurate known-groups discrimination may be necessary, but will never be sufficient, to support the validity of scores. This article elaborates on this and other problems with the known-groups comparison that limit its utility as a source of validity evidence.
Descriptors: Expertise, Novices, Validity, Comparative Analysis, Scores, Differences, Evaluation, Medical Education
Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: firstname.lastname@example.org; Web site: http://www.springerlink.com
Publication Type: Journal Articles; Reports - Evaluative
Education Level: Higher Education; Postsecondary Education
Authoring Institution: N/A