NotesFAQContact Us
Collection
Advanced
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED510378
Record Type: Non-Journal
Publication Date: 2008-Jul
Pages: 55
Abstractor: As Provided
Reference Count: 22
ISBN: N/A
ISSN: N/A
Test Scaling and Value-Added Measurement. Working Paper 2008-23
Ballou, Dale
National Center on Performance Incentives, Paper presented at the Wisconsin Center for Education Research's National Conference on Value-Added Modeling (Jul 2008)
As currently practiced, value-added assessment relies on a strong assumption about the scales used to measure student achievement, namely that these are interval scales, with equal-sized gains at all points on the scale representing the same increment of learning. Many of the metrics in which test results are expressed do not have this property (e.g., percentile ranks, normal curve equivalents). However, this property is claimed for the scales obtained when tests are scored according to Item Response Theory. This claim requires that examinees and test items constitute, in the terminology of representational measurement theory, a conjoint structure. Unfortunately, it is difficult to confirm that this condition is met. In addition, end users of the data lack access to item-level data to test these assumptions themselves. The best they can do is to check the plausibility of the resulting scales. On this count, IRT scales often do poorly. Reasonable rescalings have a substantial impact on students' measured growth. Methods of ordinal data analysis can be employed instead, on the weaker assumption that IRT scales permit us to rank students. Instead of comparing mean achievement of a teacher's students to the students of a (hypothetical) average teacher, ordinal analysis asks what fraction of the former outperforms the latter. The feasibility of such methods for value-added analysis is demonstrated. It is seen that value-added estimates are sensitive to the choice of ordinal methods over conventional techniques. Clearly, if IRT scores are an interval-scaled variable, ordinal methods throw away valuable information. Practitioners of value-added measurement should ask themselves, however, whether they are so confident of the metric properties of these scales that they are willing to attribute differences between regression-based estimates of value added and estimates based on ordinal analysis to the superiority of the former. (Contains 5 tables, 5 figures, and 26 footnotes.)
National Center on Performance Incentives. Peabody College of Vanderbilt University, PMB #43, 230 Appleton Place, Nashville, TN 37203. Tel: 615-322-5538; Fax: 615-322-6018; e-mail: ncpi@vanderbilt.edu; Web site: http://www.performanceincentives.org
Publication Type: Reports - Descriptive; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: Vanderbilt University, National Center on Performance Incentives