To estimate the long-term effects of the Reading Recovery® intervention, a regression discontinuity design (RD) was implemented in a randomly selected sample of Reading Recovery schools during each year of the federally-funded i3 Scale-Up external evaluation (2011-2015) and also in one additional cohort during the 2016-2017 school year. Long-term…

Historically, there has been concern about students losing reading ability over extended breaks from school, commonly in the summer, but studies of this phenomenon have produced inconsistent results. We applied exploratory visual analysis of multiple datasets to examine whether students in Grades K-5 appear to lose or improve in various reading…

Providing evidence that can inform awarding is an important application of Comparative Judgement (CJ) methods in high-stakes qualifications. The process of marking scripts is not changed, but CJ methods can assist in the maintenance of standards from one series to another by informing decisions about where to place grade boundaries or cut scores.…

A sample of 108 highly gifted middle school students participated in a study of the relationships between Big Five factors and overexcitabilities. Students completed the NEO-FFI and Overexcitabilities Questionnaire-II (OEQ-II). A cutoff score applied to the OEQ-II created a threshold for overexcitability, ensuring only extreme responses. Analysis…

This study used standards setting to establish cutscores for the fidelity of implementation of an evidence-based intervention, the Self-Determined Learning Model of Instruction, designed to enhance goal-directed actions in secondary students with and without disabilities. Cutscores were then applied to a fidelity data from a large, randomized…

Variation in examiner stringency is an ongoing problem in many performance settings such as in OSCEs, and usually is conceptualised and measured based on scores/grades examiners award. Under borderline regression, the standard within a station is set using checklist/domain scores and global grades acting in combination. This complexity requires a…

Regression discontinuity (RD) designs are commonly used for program evaluation with continuous treatment assignment variables. But in practice, treatment assignment is frequently based on ordinal variables. In this study, we propose an RD design with an ordinal running variable to assess the effects of extended time accommodations (ETA) for…

In this research report, we describe a study to map the scores of the "TOEFL® Essentials"™ test to the Canadian Language Benchmarks (CLB). The TOEFL Essentials test is a four-skills assessment of foundational English language skills and communication abilities in academic and general (daily life) contexts. At the time of writing this…

Interest in telehealth assessment for autism has increased due to COVID-19 and subsequent expansion of remote psychological services, though options that are easy for clinicians to adopt and available through the lifespan are limited. The Brief Observation of Symptoms of Autism (BOSA) provides a social context with standardized materials and…

Despite the increasing demand for diagnostic information, observed subscores have been often reported to lack adequate psychometric qualities such as reliability, distinctiveness, and validity. Therefore, several statistical techniques based on CTT and IRT frameworks have been proposed to improve the quality of subscores. More recently, DCM has…

Racial disparities exist in autism diagnosis, and yet, the development of most diagnostic tools has not explicitly examined measurement equity between racial and ethnic groups. We examined the validity of the Toddler Autism Symptom Inventory (TASI), a semi-structured interview developed for diagnosis of toddlers, in non-Hispanic Black/African…

This paper studies the causal impacts of public universities on the outcomes of their marginally admitted students. I use administrative admission records spanning all 35 public universities in Texas, which collectively enroll 10 percent of American public university students, to systematically identify and employ decentralized cutoffs in SAT/ACT…

The current study aims to evaluate the performance of three non-IRT procedures (i.e., normal approximation, Livingston-Lewis, and compound multinomial) for estimating classification indices when the observed score distribution shows atypical patterns: (a) bimodality, (b) structural (i.e., systematic) bumpiness, or (c) structural zeros (i.e., no…

In this digital ITEMS module, Dr. Michael Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, and body of work) but on the practical tasks that must be completed for any standard setting activity. Dr. Bunch carries the…

Various approaches help determine students' need for language support courses. Some programs use standardized language proficiency test scores, which are available from the admissions process; others use local placement tests, which are typically designed to be aligned with local language needs and ESL course curricula. Using local placement tests…