Aggregate scoring was applied to a recertifying examination for medical professionals to generate an answer key and allow comparison of peer examinees. Results for 1,927 candidates for recertification indicate considerable agreement between the traditional answer key and the aggregate answer key. (TJH)

Effects of different weighting schemes on selecting the optimal number of observations in multivariate-multifacet generalizability designs are studied when cost constraints are imposed. Comparison of four schemes through simulation indicates that all four produce similar optimal values and that reliability should be similar. (SLD)

An efficient search procedure is presented for determining the optimal number of observations of facets in a design that maximize generalizability when resource constraints are imposed. The procedure is illustrated for three-facet and four-facet designs, with extensions for other configurations. (Author/SLD)

The implications of serially correlated effects on the results of generalizability analyses are discussed. Simulated data are provided that demonstrate the biases that serially correlated effects introduce into the results. Serial correlation in measurement effects can have a marked influence on the impression of the dependability of measurement…

Using data from an individually administered interview schedule (the Consumer Satisfaction Inventory), reliability among nine interviewers was estimated with several statistical methods, including simple percentages of agreement, kappa and weighted kappa, Pearson correlations, t tests on interviewers' means, and generalizability theory techniques.…

Used generalizability theory to investigate the impact of the number of raters and the type of decision (relative versus absolute) on the reliability of writing scores. Results from 251 middle school students and 20 intermediate grade students show that reliability coefficients decline as the number of raters declines and when absolute decisions…

Addresses generalizability theory (GT), which offers a flexible framework for assessing dependability of measurement. GT allows for consideration of multiple sources of error, allowing investigators to assess the overall impact of measurement error. Illustrative analyses demonstrate the special advantages of GT for planning studies in which…

Ratings of test item relevance were collected from 57 practitioners from a pretest of a medical certifying examination. Ratings were correlated with item difficulty, but the relationship between ratings and item discrimination was less clear. Application of generalizability theory shows that reasonable estimates of item, stem, and total test…

Little research exists on the assessment of administrators' performance in higher education. The authors offer an evaluation model for assessing and monitoring the effectiveness of academic deans and directors, using generalizability theory as a basis for developing more accurate assessment procedures. (JM)

Value-added models in education research allow researchers to explore how a wide variety of policies and measured school inputs affect the academic performance of students. Researchers typically quantify the impacts of such interventions in terms of "effect sizes", i.e., the estimated effect of a one standard deviation change in the…

In this study, we discuss the use of generative evaluation as an innovative approach in the analysis of qualitative interview data for evaluating simultaneous renewal of Iowa State University's PT3 grant. Goodlad's (1994) simultaneous renewal framework forms the basis of our argument. We focused on the CREATER+ model designed to explain the…

In this study, we discuss the use of generative evaluation as an innovative approach in the analysis of qualitative interview data for evaluating simultaneous renewal of Iowa State University's PT3 grant. Goodlad's (1994) simultaneous renewal framework forms the basis of our argument. We focused on the CREATER+ model designed to explain the…

The focus of the present study is on students' experiences of enjoyment, an emotion largely neglected in educational research. We present a model in which specific levels of generalization of the construct of enjoyment are differentiated. Based on their extent of generalization, these differentiated constructs of enjoyment are located in a…

Although multivariate generalizability theory was developed more than 30 years ago, little published research utilizing this framework exists and most of what does exist examines tests built from tables of specifications. In this context, it is assumed that the universe scores from levels of the fixed multivariate facet will be correlated, but the…

Statistical invariance procedures provide a way of looking at the generalizability of research results from sample to sample when the research has not been validated by replication. This paper discusses the Procrustean Rotation invariance procedure following a canonical correlation analysis. The computer program RELATE is used to gauge the…