NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1036023
Record Type: Journal
Publication Date: 2014-May
Pages: 18
Abstractor: As Provided
Reference Count: 45
ISSN: ISSN-1382-4996
What Counts as Validity Evidence? Examples and Prevalence in a Systematic Review of Simulation-Based Assessment
Cook, David A.; Zendejas, Benjamin; Hamstra, Stanley J.; Hatala, Rose; Brydges, Ryan
Advances in Health Sciences Education, v19 n2 p233-250 May 2014
Ongoing transformations in health professions education underscore the need for valid and reliable assessment. The current standard for assessment validation requires evidence from five sources: content, response process, internal structure, relations with other variables, and consequences. However, researchers remain uncertain regarding the types of data that contribute to each evidence source. We sought to enumerate the validity evidence sources and supporting data elements for assessments using technology-enhanced simulation. We conducted a systematic literature search including MEDLINE, ERIC, and Scopus through May 2011. We included original research that evaluated the validity of simulation-based assessment scores using two or more evidence sources. Working in duplicate, we abstracted information on the prevalence of each evidence source and the underlying data elements. Among 217 eligible studies only six (3%) referenced the five-source framework, and 51 (24%) made no reference to any validity framework. The most common evidence sources and data elements were: relations with other variables (94% of studies; reported most often as variation in simulator scores across training levels), internal structure (76%; supported by reliability data or item analysis), and content (63%; reported as expert panels or modification of existing instruments). Evidence of response process and consequences were each present in <10% of studies. We conclude that relations with training level appear to be overrepresented in this field, while evidence of consequences and response process are infrequently reported. Validation science will be improved as educators use established frameworks to collect and interpret evidence from the full spectrum of possible sources and elements.
Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail:; Web site:
Publication Type: Journal Articles; Information Analyses; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A