ERIC Number: ED441839
Record Type: RIE
Publication Date: 2000-Apr
A Construct-Centered Generalizability Model: Analyzing Underlying Constructs of Cognitively Complex Performance Assessments.
Jiang, Ying Hong; Smith, Philip L.
With a construct-centered reliability analytical approach the reliability analysis should crystallize the multi-traits or constructs that the test specialists developed to measure from student performance and then estimate the degree of fit between the theoretical expectations of test developers and the performance exhibited by students. This study attempts to provide an empirical example of a construct-centered reliability analysis using writing performance assessment data from a large urban school district. Writing assessments from 17,330 students in grades 3, 5, and 8 were used in the analysis. Scoring guides were developed, centering the constructs comprised of dimensions for raters to consider when scoring. Resampling was done by randomly selecting three of four percent of the students from the assessments before a generalizability study with a fully crossed two-facet design was conducted. A large proportion of variance was estimated due to the constructs of rhetorical effectiveness and conventions, as well as the interaction between constructs and raters. Variance due to rater facet was very small. An array of acceptable G coefficients across samples was obtained. It is suggested that high reliability or generalizability is achievable using a construct-centered reliability approach to identify construct relevant variances when there is a high degree of fit between the substantive expectations generated from the test specialists' understanding of the construct as realized by the measurement procedure and observation. (Contains 7 tables, 12 charts, and 10 references.) (SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Note: Paper presented at the Annual Meeting of the American Educational Research Association (New Orleans, LA, April 24-28, 2000).