Within classrooms, psychologists and teachers use direct behavior observation methods, systematic behavior observations (SBOs) and direct behavior ratings (DBRs), to gather information about students' behaviors for the purposes of making decisions related to diagnosis and classroom management or behavioral feedback respectively. Observers use SBOs…

In learning diagnostic assessments, the attribute hierarchy specifies a sequential network of interrelated attribute mastery processes, which makes a test blueprint consistent with the cognitive theory. One of the most important functions of attribute hierarchy is to guide or limit the developmental direction of students and then form a…

Adaptive tests are more efficient than fixed-length tests through the use of item response theory; adaptive tests also present students questions that are tailored to their proficiency level. Although the adaptive algorithm is straightforward, developing a multidimensional computer adaptive test (MCAT) measure is complex. Evidence-centered design…

Module Overview: In this digital ITEMS module, Dr. Jodi M. Casabianca provides a primer on the "hierarchical rater model" (HRM) framework and the recent expansions to the model for analyzing raters and ratings of constructed responses. In the first part of the module, she establishes an understanding of the nature of constructed…

In this article we address the mounting criticism and rejection of standardized tests used in the selection of students for college or university education. Admission tests are being increasingly demonized in many parts of the world and many colleges and universities are dropping tests for selection purposes, claiming the tests are detrimental to…

Commonly used machine learning applications seem to relate to big data. This article provides a gentle review of machine learning and shows why machine learning can be applied to small data too. An example of applying machine learning to screen irregularity reports is presented. In the example, the support vector machine and multinomial naïve…

In this study, we compared the results of 27 special education teachers' evaluations using two different observation instruments, the Framework for Teaching (FFT), and the Explicit Instruction observation protocol of the Recognizing Effective Special Education Teachers (RESET) observation system. Results indicate differences in the rank-ordering…

Refined tracking allows students and teachers to more accurately understand students' learning growth. To provide refined learning tracking with longitudinal diagnostic assessment, this article proposed a new model by incorporating probabilistic logic into longitudinal diagnostic modeling. Specifically, probabilistic attributes were used instead…

The continuous testing framework, where both successful and unsuccessful examinees have to demonstrate continued proficiency at frequent prespecified intervals, is a framework that is used in noncognitive assessment and is gaining in popularity in cognitive assessment. Despite the rigorous advantages of this framework, this paper demonstrates that…

The community of educational measurement researchers and practitioners has made many positive contributions to education, but has also become complacent and lost the public trust. In this article, reasons for the lack of public trust in educational testing are described, and core values for educational measurement are proposed. Reasons for…

Technical difficulties occasionally lead to missing item scores and hence to incomplete data on computerized tests. It is not straightforward to report scores to the examinees whose data are incomplete due to technical difficulties. Such reporting essentially involves imputation of missing scores. In this paper, a simulation study based on data…

Large-scale assessments such as the Programme for International Student Assessment (PISA) have field trials where new survey features are tested for utility in the main survey. Because of resource constraints, there is a trade-off between how much of the sample can be used to test new survey features and how much can be used for the initial item…

Despite the ease of accessing a wide range of measures, little attention is given to validity arguments when considering whether to use the measure for a new purpose or in a different context. Making a validity argument has historically focused on the intended interpretation and use. There has been a press to consider both the intended and actual…

For nearly three decades, researchers have been concerned that the educational measurement field is not producing enough graduate-level specialists to meet the growing demand driven by the increased use of educational assessments in the United States. This study examined the supply-side aspect of the proposed labor shortage by relying on data from…

Subscore added value analyses assume invariance across test taking populations; however, this assumption may be untenable in practice as differential subdomain relationships may be present among subgroups. The purpose of this simulation study was to understand the conditions associated with subscore added value noninvariance when manipulating: (1)…