ERIC Number: ED408340
Record Type: Non-Journal
Publication Date: 1997
Reference Count: N/A
Estimating Rater Severity with Multilevel and Multidimensional Item Response Modeling.
Traditional approaches to the investigation of the objectivity of ratings for constructed-response items are based on classical test theory, which is item-dependent and sample-dependent. Item response theory overcomes this drawback by decomposing item difficulties into genuine difficulties and rater severity. In so doing, objectivity of ability estimates is achieved, even though objectivity of ratings is poor. However, most item response models are too rigid to fit complexity of rater severities. Also, other types of items in the same test are excluded when estimating rater severities. These problems are addressed in this study. Several advanced models are proposed to explore severity changes over items and within items. In addition, multilevel and multidimensional models are formed to incorporate both multiple-choice items and constructed-response items in the test to increase estimating accuracy and model fit. The proposed models are made possible by a newly developed item response model, the multidimensional and multilevel random coefficients multinomial logit model. A real data set from the biology subject of the 1995 Joint College Entrance Examination in Taiwan was analyzed to demonstrate the advantages of this approach. (Contains 5 tables, 6 figures, and 35 references.) (Author)
Publication Type: Reports - Evaluative
Education Level: N/A
Sponsor: Taiwan National Science Council, Taipei.
Authoring Institution: N/A
Identifiers - Location: Taiwan