This article describes two separate, related studies that provide insight into the effectiveness of "e-rater" score calibration methods based on different distributional targets. In the first study, we developed and evaluated a new type of "e-rater" scoring model that was cost-effective and applicable under conditions of absent human rating and…

Studies of international mathematics achievement such as the Trends in Mathematics and Science Study (TIMSS) have employed classical test theory and item response theory to rank individuals within a latent ability continuum. Although these approaches have provided insights into comparisons between countries, they have yet to examine how specific…

The aim of this study was to investigate response-process based evidence for the validity of the Programme for International Student Assessment's (PISA) self-report questionnaire scales as measures of specific psychological constructs, with a focus on scales meant to measure inclination toward specific learning strategies. Cognitive interviews (N…

Measurement invariance of the 8-factor Inventory of School Motivation (McInerney & Sinclair, 1991) between American and Chinese college students was tested using single-group and multi-group confirmatory factor analysis. A Mandarin Chinese version of the ISM was developed for this study. Comparisons of latent means were conducted when warranted by…

The intent of this study was to examine the views of teachers regarding the appropriateness of the purposes and uses of the provincial assessments in Alberta and Ontario and the seriousness of the concerns raised about these assessments. These provinces represent educational jurisdictions that use large-scale assessments within a low-stakes…

This study examined the effect of including or excluding repeaters on the equating process and results. New forms of two tests were equated to their respective old forms using either all examinees or only the first timer examinees in the new form sample. Results showed that for both tests used in this study, including or excluding repeaters in the…

This article delves into a relatively unexplored area of measurement by focusing on adaptive testing with unidimensional pairwise preference items. The use of such tests is becoming more common in applied non-cognitive assessment because research suggests that this format may help to reduce certain types of rater error and response sets commonly…

The current study examined whether psychological reactance differs across compliant and non-compliant examinees. Given the lack of consensus regarding the factor structure and scoring of the Hong Psychological Reactance Scale (HPRS), its factor structure was evaluated and subsequently tested for measurement invariance (configural, metric, and…

The aims of this study are (a) to examine the sources of differential functioning by gender via differential bundle functioning (DBF) in mathematics assessment and (b) to use DBF to explore whether the differential functioning displayed is construct-relevant or construct-irrelevant. Three qualitatively different areas, namely curriculum domains,…

The equal ability distribution assumption associated with the equivalent groups equating design was investigated in the context of a selection test for admission to higher education. The purpose was to assess the consequences for the test-takers in terms of receiving improperly high or low scores compared to their peers, and to find strong…

We used multiple regression to calculate a new Ego Impairment Index (EII-3). The aim was to incorporate changes in the component variables and distribution of the number of responses as found in the new Rorschach Performance Assessment System, while sustaining the validity and reliability of previous EIIs. The EII-3 formula was derived from a…

After a test is developed, most content validation analyses shift from ascertaining domain definition to studying domain representation and relevance because the domain is assumed to be set once a test exists. We present an approach that allows for the examination of alternative domain structures based on extant test items. In our example based on…

We present an extension of the logistic regression procedure to identify dichotomous differential item functioning (DIF) in the presence of more than two groups of respondents. Starting from the usual framework of a single focal group, we propose a general approach to estimate the item response functions in each group and to test for the presence…

This study examined test takers' perception of assessment demand and its impact on the measurement of intended constructs. More than 800 test takers took a pre- and a posttest of College English Test Band 4 and filled in a perception questionnaire to report the skills they perceive as necessary for answering the test. The study found test takers…

Throughout the world, cut scores are an important aspect of a high-stakes testing program because they are a key operational component of the interpretation of test scores. One method for setting standards that is prevalent in educational testing programs--the Bookmark method--is intended to be a less cognitively complex alternative to methods…