This article proposes a Boolean approach to representing and analyzing interobserver agreement in dichotomous coding. Building on the notion that observations are samples of a universe of observations, it submits that coding can be viewed as a process in which observers sample pieces of evidence on constructs. It distinguishes between formal and…

We investigated whether consensus-based test translation review procedures can be used effectively in cultural contexts with high social stratification. We staged two test translation review panels in China -- where social stratification may potentially inhibit individuals' ability to express opinion and disagreement. We adapted a consensus-based…

We address the concern that classification of English learners (ELs) primarily based on standardized tests does not accurately reflect what students know and can do with English. While qualitative classroom discourse analyses show that ELs' language-in-use is not as limited as test scores indicate, available evidence is sparse and prone to…

We examined semiotic exchangeability in pop-up glossary translations and illustrations used as supports for second language learners (SLLs) in computer-administered mathematics tests. In a sample of 516 mathematics items, Grades 3-8 and 11, from a large-scale assessment program in the US, test developers identified terms that could be translated…

This paper is concerned with fair and valid testing. It reports on a long-term project aimed at creating visual accessibility resources in mathematics computer-administered test items in a large-scale assessment system. We discuss our experience designing and developing pop-up illustration glossaries (IGs) for second language learners (SLLs) --…

We investigated how high school mathematics course placement contributes to inequalities in college preparation for students categorized as "English Learners" (ELs), especially "long-term" English Learners (LTELs). We devised a base-2 enumeration approach that allowed us to process complex transcript data from one school…

This article addresses test translation review in international test comparisons. We investigated the applicability of the theory of test translation error--a theory of the multidimensionality and inevitability of test translation error--across source language-target language combinations in the translation of PISA (Programme of International…

We examined multimodality (the representation of information in multiple semiotic modes) in the context of international test comparisons. Using Program of International Student Assessment (PISA)-2009 data, we examined the correlation of the difficulty of science items and the complexity of their illustrations. We observed statistically…