The purpose of this instructional piece was to provide a nontechnical synthesis of common internal consistency reliability estimates used in professional counseling and in related fields. The article begins with an overview of coefficients alpha, omega, omega hierarchical, and H, with guidelines for their selection. Next, I provide recommendations…

Recent advances in automated scoring technology have made it practical to replace multiple-choice questions (MCQs) with short-answer questions (SAQs) in large-scale, high-stakes assessments. However, most previous research comparing these formats has used small examinee samples testing under low-stakes conditions. Additionally, previous studies…

Test developers and psychometricians have historically examined measurement bias and differential item functioning (DIF) across a single categorical variable (e.g., gender), independently of other variables (e.g., race, age, etc.). This is problematic when more complex forms of measurement bias may adversely affect test responses and, ultimately,…

This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. We investigated…

In many large-scale testing programs, examinees are frequently categorized into different performance levels. These classifications are then used to make high-stakes decisions about examinees in contexts such as in licensure, certification, and educational assessments. Numerous approaches to estimating the consistency and accuracy of this…

Computerized adaptive testing (CAT) has gained deserved popularity in the administration of educational and professional assessments, but continues to face test security challenges. To ensure sustained quality assurance and testing integrity, it is imperative to establish and maintain multiple stable item pools that are consistent in terms of…

The purpose of this study was to examine variation in assistant principal (AP) social justice beliefs and high-stakes accountability perceptions and explore how APs balance accountability policies while addressing social justice issues in their schools. 79 assistant principals participated in a phase 1 survey with 10 APs selected to participate in…

Test scores are often used to make decisions about examinees, such as in licensure and certification testing, as well as in many educational contexts. In some cases, these decisions are based upon compensatory scores, such as those from multiple sections or components of an exam. Classification accuracy and classification consistency are two…

Suitable execution of moderation policy is challenging but crucial for the trustworthiness and credibility of internal high-stakes assessment systems. In formal education, policies are rarely implemented as intended. Instead, they are "enacted" in ways influenced by mediating factors including the internal and external contexts of…

New public management ideals and standards have become increasingly adhered to in health professions education; this is particularly apparent in high-stakes assessment, as a gateway to practice. Using an Institutional Ethnographic approach, we looked at the work involved in running high-stakes Objective Structured Clinical Exams (OSCEs) throughout…

This article considers the challenges of using artificial intelligence (AI) and machine learning (ML) to assist high-stakes standardised assessment. It focuses on the detrimental effect that even state-of-the-art AI and ML systems could have on the validity of national exams of secondary education, and how lower validity would negatively affect…

Differentiation is an instructional practice teachers employ to modify their classroom content, process, and products based on student readiness, interest, and learning profile. Many school districts recognize the benefits of differentiated instruction and thus mandate allotted classroom time for its implementation. In this paper, we investigate…

Administrative problems such as computer malfunction and power outage occasionally lead to missing item scores, and hence to incomplete data, on credentialing tests such as the United States Medical Licensing examination. Feinberg compared four approaches for reporting pass-fail decisions to the examinees with incomplete data on credentialing…

This paper explores students' preparatory processes for high-stakes exams using Danish oral exams as an example. To graduate, students must convince two teacher-examiners as the state's representatives that they deserve to pass. Average grades determine students' admission into tertiary education. Fieldwork data following students transitioning…

Objective structured clinical examination (OSCE) is widely used to assess medical students' clinical skills. Virtual OSCEs were used in place of in-person OSCEs during the COVID-19 pandemic; however, their reliability is yet to be robustly analyzed. By applying generalizability (G) theory, this study aimed to evaluate the reliability of a hybrid…