ERIC Number: ED414281
Record Type: Non-Journal
Publication Date: 1997-Mar
Reference Count: N/A
The Generalizability of Scoring TIMSS Open-Ended Items.
Smith, Teresa A.
The Third International Mathematics and Science Study (TIMSS) measured mathematics and science achievement of middle school students in more than 40 countries. About one quarter of the tests' nearly 300 items were free response items requiring students to generate their own answers. Scoring these responses used a two-digit diagnostic code rubric with the first digit determining the correctness of the response and the second used to identify certain types of responses showing common approaches or misconceptions. This paper discusses relative contributions of the sources of error variance in student and country-level scores on these open-ended items due to rater effects as a function of item type. Fifty student booklets from 7 English speaking countries were analyzed for 350 student responses for each item. Generalizability studies determined the variability of scores associated with effects due to raters. Generalizability coefficients indicated a high degree of reliability in the relative rankings of a country's average score on TIMSS free-response ratings based on data from the cross-country coding study. The generalizability for an individual's score on a particular item was found to be somewhat less stable for some items, but this was not a concern, since the purpose of the TIMSS was to report country-level averages. (Contains six tables and five figures.) (SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Trends in International Mathematics and Science Study