The practice of screening students to identify behavioral and emotional risk is gaining momentum, with limited guidance regarding the frequency with which screenings should occur. Screening frequency decisions are influenced by the stability of the constructs assessed and changes in risk status over time. This study investigated the 4-year…

Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental…

The many-faceted Rasch (MFR) model has been used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR model for examining the quality of ratings obtained from a standard…

This study integrated Bayesian hierarchical modeling and receiver operating characteristic analysis (BROCA) to evaluate how interest strength (IS) and interest differentiation (ID) predicted low–socioeconomic status (SES) youth's interest-major congruence (IMC). Using large-scale Kuder Career Search online-assessment data, this study fit three…

The current study evaluated the relationship between various operationalizations of the Advanced Placement[R] (AP) exam and course information with first-year grade point average (FYGPA) in college to better understand the role of AP in college admission decisions. In particular, the incremental validity of the different AP variables, above…

The study aimed to develop the Homophobic Bullying Scale and to investigate its psychometric properties. The items of the Homophobic Bullying Scale were created to measure high school students' bullying behaviors motivated by homophobia, including verbal bullying, relational bullying, physical bullying, property bullying, sexual harassment, and…

This study examined the amount of time that different ability-level examinees spend on questions they answer correctly or incorrectly across different pretest item blocks presented on a fixed-length, time-restricted computerized adaptive testing (CAT). Results indicate that different ability-level examinees require different amounts of time to…

Whether an ability test delivered on either paper or computer provides the same information is an important question in applied psychometrics. Besides the validity, it is also the fairness of a measure that is at stake if the test medium affects performance. This study provides a comprehensive review of existing equivalence research in the field…

Quantitative literacy is a habit of mind that is characterized by the interrelationship among a person's everyday understanding of mathematics, his or her beliefs about mathematics, and his or her disposition toward mathematics. To assess quantitative literacy, it is important to devise measurement tools that provide valid and reliable information…

The purpose of this validity study was to improve measurement of athletes' evaluations of their head coach's coaching competency, an important multidimensional construct in models of coaching effectiveness. A revised version of the Coaching Competency Scale (CCS) was developed for athletes of high school teams (APCCS II-HST). Data were collected…

The purpose of this study is to test the validity of scores on the Homework Purpose Scale using 681 rural and 306 urban high school students. First, confirmatory factor analysis was conducted on the rural sample. The results reveal that the Homework Purpose Scale comprises three separate yet related factors, including Learning-Oriented Reasons,…

Three studies were conducted to develop and validate scores on a new measure appropriate for assessing adolescents' situational interest (SI) across various academic settings. In Study 1 (n = 858), a self-report questionnaire was administered to undergraduates in introductory psychology. Confirmatory factor analyses (CFA) supported a three-factor…

Statistical biases associated with single-level analyses underscore the importance of partitioning variance/covariance matrices into individual and group levels. From a multilevel perspective based on data from 21,579 students in 58 high schools, the present study assesses the multilevel factor structure of motivation and engagement with a…

The aim of this study was to develop a scale assessing high school students' self-efficacy beliefs in chemistry-related tasks and to assess psychometric properties of scores on this scale. A pilot study with a sample of 150 high school students provided initial evidence for two-factor structure of 16-item scale, named High School Chemistry…

This study investigates the effect of mode of administration of the Raven Standard Progressive Matrices test on distribution, accuracy, and meaning of raw scores. A random sample of high school students take counterbalanced paper-and-pencil and computer-based administrations of the test and answer a questionnaire surveying preferences for…