In this note, we demonstrate an interesting use of the posterior distributions (and corresponding posterior samples of proficiency) that are yielded by fitting a fully Bayesian test scoring model to a complex assessment. Specifically, we examine the efficacy of the test in combination with the specific passing score that was chosen through expert…

Modified the three-parameter model to include an additional random effect for items nested within the same testlet. Fitted the new model to 86 testlets from the Test of English as a Foreign Language (TOEFL) and compared standard parameters (discrimination, difficulty, and guessing) with those obtained through traditional modeling. Discusses the…

Describes an analytic method for aiding in the generation of subscores that characterize the deep structure of tests and derives a procedure for estimating scores for those scales that are more statistically stable than subscores composed solely of items contained on that scale. Used data from a Praxis administration (9,278 examinees) to show the…

Redesigned five displays for communicating results from the National Assessment of Educational Progress (NAEP) 1994 reading assessment using principles developed by H. Wainer (1997). Responses from 10 educators show more accurate and faster responses to questions asked after the redesign. Discusses implications for information communication in…

A trilinear chart for understanding the distribution of achievement levels of students' performance in the National Assessment of Educational Progress (NAEP) is described and its use illustrated with data from the 1992 state NAEP mathematics assessment. Trends for all 44 participating jurisdictions are simultaneously displayed. (SLD)

Describes recent research attempts to draw inferences about the relative standing of the states on the basis of mean SAT scores. This paper identifies five serious errors that call into question the validity of such inferences. Some plausible ways to avoid the errors are described. (Author/LMO)

An example demonstrates and explains that summary statistics commonly used to measure test quality can be seriously misleading and that summary statistics for the whole test are not sufficient for judging the quality of the test. (Author/LMO)

Techniques of exploratory data analysis (EDA) were used to decompose data tables portraying performance of ethnic groups on the Scholastic Aptitude Test. These analyses indicate the size and structure of differences in performance among groups studied, nature of changes across time, and interactions between group membership and time. (Author/DWH)

The testlet, a bundle of test items, alleviates some problems associated with computerized adaptive testing: context effects, lack of robustness, and item difficulty ordering. While testlets may be linear or hierarchical, the most useful ones are four-level hierarchical units, containing 15 items and partitioning examinees into 16 classes. (GDC)

This paper reviews the role of the item in test construction, and suggests some new methods of item analysis. A look at dynamic, graphical item analysis is provided that uses the advantages of modern, high-speed, highly interactive computing. Several illustrations are provided. (Author/TJH)

Three different applications of the testlet concept are presented, and the psychometric models most suitable for each application are described. Difficulties that testlets can help overcome include (1) context effects; (2) item ordering; and (3) content balancing. Implications for test construction are discussed. (SLD)

Hierarchical (adaptive) and linear methods of testlet construction were compared. The performance of 2,080 ninth and tenth graders on a 4-item testlet was used to predict performance on the entire test. The adaptive test was slightly superior as a predictor, but the cost of obtaining that superiority was considerable. (SLD)

Computer simulations were run to measure the relationship between testlet validity and factors of item pool size and testlet length for both adaptive and linearly constructed testlets. Making a testlet adaptive yields only modest increases in aggregate validity because of the peakedness of the typical proficiency distribution. (Author/SLD)

Focusing on educational measurement that suggests an action and has an outcome, 16 problem areas are defined and grouped into the following classes: (1) validity; (2) issues of statistical adjustment; (3) data insufficiencies; (4) other issues related to standardized testing and constructed responses; and (5) technical issues of psychometrics.…

A testlet is an integrated group of test items presented as a unit. The concept of testlet differential item functioning (testlet DIF) is defined, and a statistical method is presented to detect testlet DIF. Data from a testlet-based experimental version of the Scholastic Aptitude Test illustrate the methodology. (SLD)