Assessing the comparability of different groups is an issue facing many researchers and evaluators in a variety of settings. Commonly, null hypothesis significance testing (NHST) is incorrectly used to demonstrate comparability when a non-significant result is found. This is problematic because a failure to find a difference between groups is not…

In developing attitudinal instruments for young children, researchers, program evaluators, and clinicians often use response scales with pictures or images (e.g., smiley faces) as anchors. This article considers connections between word-based and picture based Likert scales and highlights the value in translating conventions used in word-based…

When respondents answer paper-and-pencil (PP) questionnaires, they sometimes modify their responses to correspond to previously answered items. As a result, this response bias might artificially inflate the reliability of PP questionnaires. We compared the internal consistency of PP questionnaires to computerized questionnaires that presented a…

Standardized effect size measures typically employed in behavioral and social sciences research in the multi-group case (e.g., [eta][superscript 2], f[superscript 2]) evaluate between-group variability in terms of either total or within-group variability, such as variance or standard deviation--that is, measures of dispersion about the mean. In…

This paper introduces a simple and intuitive graphical display for transition table based accountability models that can be used to communicate information about students' status and growth simultaneously. This graphical transition table includes the use of shading to convey year to year transitions and different sized letters for performance…

Large surveys often use probability sampling in order to obtain representative samples, and these data sets are valuable tools for researchers in all areas of science. Yet many researchers are not formally prepared to appropriately utilize these resources. Indeed, users of one popular dataset were generally found "not" to have modeled the analyses…

This study discusses and presents an example of a use of spline functions to establish and report test scores using a moderated system of any number of cut scores. Our main goals include studying the need for and establishing moderated standards and creating a reporting scale that is referenced to all the standards. Our secondary goals are to make…

This study shows the extent to which effect size is reported and discussed in four major journals. A series of judgments about different aspects of effect size were conducted for 417 articles from four journals. Results suggest that while the reporting of simple effect size indices is more prevalent, substantive discussions of the meaning of…

There are relatively few examples of quantitative approaches to quality control in educational assessment and accountability contexts. Among the several techniques that are used in other fields, Shewart charts have been found in a few instances to be applicable in educational settings. This paper describes Shewart charts and gives examples of how…

Recent studies report that the use of rubrics may not improve the reliability of assessment if raters are not well trained on how to design and employ them effectively. The intent of this two-phase study was to test if training pre-service and new in-service teachers in the construction, use, and evaluation of rubrics would improve the reliability…

This research replicates the work of Starch and Elliot (1912) by examining the reliability of the grading by English teachers in a single school district. Ninety high school teachers graded the same student paper following professional development sessions in which they were trained to use NWREL's "6+1 Traits of Writing." These participants had…

At the same time as some faculty committees and corporations are appealing to the use of online ratings from RateMyProfessors.com to inform promotion decisions and nationwide university rankings, others are derogating the site as an unreliable source of idiosyncratic student ratings and commentary. In this paper we describe a study designed to…

Structured questionnaires and semi-structured interviews are often used in mixed method studies to generate confirmatory results despite differences in methods of data collection, analysis, and interpretation. A review of 19 questionnaire-interview comparison studies found that consensus and consistency statistics were generally weak between…

The first part of this paper describes the use of the robust z[subscript R] statistic to link test forms using the Rasch (or one-parameter logistic) model. The procedure is then extended to the two-parameter and three-parameter logistic and two-parameter partial credit (2PPC) models. A real set of data was used to illustrate the extension. The…

Cognitive diagnostic modeling has become an exciting new field of psychometric research. These models aim to diagnose examinees' mastery status of a group of discretely defined skills, or attributes, thereby providing them with detailed information regarding their specific strengths and weaknesses. Combining cognitive diagnosis with computer…