NotesFAQContact Us
Search Tips
ERIC Number: ED442857
Record Type: Non-Journal
Publication Date: 2000-Apr
Pages: 43
Abstractor: N/A
Reference Count: N/A
A Subdividing Method for Generalizability Theory: Precision of Measurement Errors and Patterns of Missing Data.
Chiu, Christopher W. T.
A procedure was developed to analyze data with missing observations by extracting data from a sparsely filled data matrix into analyzable smaller subsets of data. This subdividing method, based on the conceptual framework of meta-analysis, was accomplished by creating data sets that exhibit structural designs and then pooling variance components from these designs. A Monte Carlo simulation was used to examine the statistical properties of the variance-component estimates and some commonly used composite indices, the generalizability coefficient and the dependability coefficient. Data sets used to evaluate the method ranged from 750 examinees and 4 raters to 6,000 examinees and 28 rates with 2 tasks. Experimental conditions, such as item difficulty and rater inconsistency, were varied to model operational performance assessments. The subdividing method recovered variance component estimates with high accuracy and precision in a variety of conditions. Increasing the number of examinees scored by the same raters from 12 to 24 had virtually no effect on the accuracy and precision of the estimates. Results indicate that the subdividing method produces outcomes having properties that are similar to those of complete data methods. Different rules for forming groups of rates changed the structural design of scores and thus influenced the precision of measurement error estimations. Results from this study indicate that scoring centers can determine and forecast how confidently they can interpret generalizability analyses by controlling the rules used to assign tasks to raters and the universe of generalization for which normative and criterion-referenced decisions are made. Appendixes contain discussions of the structure of a modified balanced incomplete block design, a model to determine the size of a rater pool, and equations for the standard error of variance components in a two-facet crossed design. (Contains 13 figures and 31 references.) (SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A