ERIC Number: ED408326
Record Type: Non-Journal
Publication Date: 1997-Mar
Reference Count: N/A
Generalizability Theory: A New Approach To Analyze Non-Crossed Performance Assessment Data.
Chiu, Chris W. T.; Wolfe, Edward W.
Unstable, and potentially invalid, variance component estimates may result from using only a limited portion of available data from operational performance assessments. However, missing observations are common in these settings because of the nature of the assessment design. This paper describes a procedure for overcoming the computational and technological limitations in analyzing data with missing observations by extracting data from a sparsely filled data set into analyzable smaller subsets of data. This parsing is accomplished by creating data sets that exhibit structural designs that are common in generalizability analyses, namely the crossed, mixed, and nested designs. An example of how to perform the procedure is given. Data are from a large-scale college writing assessment in which each of 5,905 examinees responded to 2 essay prompts. Results show that the sparsely filled performance assessment data sets can be restructured into analyzable smaller subsets of data. Results suggest that the crossed, mixed, and nested methods are comparable, but more study is needed to determine whether the methods generalize to other data sets with more than two facets. (Contains 3 figures, 9 tables, and 17 references.) (Author/SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Sponsor: American Coll. Testing Program, Iowa City, IA.
Authoring Institution: N/A