ERIC Number: EJ1048933
Record Type: Journal
Publication Date: 2015
Abstractor: As Provided
Impact of Design Effects in Large-Scale District and State Assessments
Phillips, Gary W.
Applied Measurement in Education, v28 n1 p33-47 2015
This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling errors, in turn, results in unanticipated instability in the testing program and an increase in Type I errors in significance tests. This is especially true when the standard error of equating is underestimated. The problem is caused by the typical district and state practice of using nonprobability cluster-sampling procedures, such as convenience, purposeful, and quota sampling, then calculating statistics and standard errors as if the samples were simple random samples.
Descriptors: State Programs, Sampling, Research Design, Error of Measurement, Testing Programs, Statistical Significance, School Districts, Probability, Sample Size, Response Rates (Questionnaires), Weighted Scores, Item Analysis, Equated Scores, Item Response Theory, Testing Problems, Evaluation Methods, Evaluation Problems, Experimenter Characteristics, Test Reliability, Test Validity, Group Testing
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A