NotesFAQContact Us
Search Tips
ERIC Number: ED509419
Record Type: Non-Journal
Publication Date: 2010-Apr
Pages: 20
Abstractor: As Provided
Reference Count: 22
A Generalizability Investigation of Cognitive Demand and Rigor Ratings of Items and Standards in an Alignment Study
Lombardi, Allison; Seburn, Mary; Conley, David; Snow, Eric
Online Submission, Paper presented at the Annual Meeting of the American Educational Research Association (Denver, CO, Apr 30-May 4, 2010)
In alignment studies, expert raters evaluate assessment items against standards and ratings are used to compute various alignment indices. Questions about rater reliability, however, are often ignored or inadequately addressed. This paper reports the results of a generalizability theory study of cognitive demand and rigor ratings of assessment items and college-readiness standards in the context of an alignment study of college admissions tests to a set of college readiness standards. Results indicate a higher level of generalizability for Math item and standard ratings than for English item and standard ratings, as well as a higher level of generalizability for cognitive demand ratings than for rigor ratings. Results also suggest that the standard of 5-6 raters in alignment studies may be insufficient for obtaining desired reliability. These findings may be used to carefully plan more robust alignment studies in the future so that higher levels of reliability across raters will be attained. (Contains 3 tables and 2 figures.)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A