ERIC Number: ED472522
Record Type: Non-Journal
Publication Date: 2002
Reference Count: N/A
The Effects of Matrix Sampling on Student Score Comparability in Constructed-Response and Multiple-Choice Assessments. Technical Guidelines for Performance Assessment.
Dings, Jonathan; Childs, Ruth; Kingston, Neal
This study examined matrix sampling of test content, the practice of giving various students in the same school differing test questions. This often-used approach to large-scale assessment allows for relatively broad coverage of the curriculum, but with fewer comparable individual student scores than a conventional test. One can be sure that scores will differ, but no known research tells how large the differences will be. The study used a quantitative research technique to study how much the comparability of individual student scores is sacrificed when tests are matrix sampled. Tests of varying lengths, comprising multiple-choice items, constructed-response items, and a combination of both were examined. Student scores on "pseudo-forms" were studied using forms of a grade 8 statewide mathematics test, with more than 4,000 students taking each of 12 test forms. Results show that matrix sampling reduced the comparability of student scores on average by about one-fifth to one-third of a standard deviation unit when a small percentage of the items were matrix sampled, and by about six-tenths to more than four-fifths of a standard deviation unit when the entire test was matrix sampled. Results varied according to the reliability of the test scores studied, and do not necessarily generalize to other test lengths or to data from other testing programs. The study illustrates a technique test developers could use to see how greatly the comparability of individual student scores is compromised by a matrix sampling test design. An appendix shows the percent of items covering content across the 12 original test forms. (Contains 8 tables and 18 references.) (SLD)
Descriptors: Junior High School Students, Junior High Schools, Performance Based Assessment, Reliability, Sampling, Scores, Test Construction, Test Content
Council of Chief State School Officers, One Massachusetts Avenue, Suite 700, Washington, DC 20001-1431 ($10). Tel: 202-408-5505; Fax: 202-408-8072. For full text: http://publications.ccsso.org
Publication Type: Reports - Research
Education Level: N/A
Sponsor: Office of Educational Research and Improvement (ED), Washington, DC.
Authoring Institution: Council of Chief State School Officers, Washington, DC.