A common question asked by researchers using regression models is, What sample size is needed for my study? While there are formulae to estimate sample sizes, their assumptions are often not met in the collected data. A more realistic approach to sample size determination requires more information such as the model of interest, strength of the…

Kingston and Nash (2011) recently presented a meta-analysis of studies showing that the effect of formative assessment on K-12 student achievement may not be as robust as widely believed. This investigation analyzes the methodology used in the Kingston and Nash meta-analysis and provides further analyses of the studies included in the study. These…

Structured questionnaires and semi-structured interviews are often used in mixed method studies to generate confirmatory results despite differences in methods of data collection, analysis, and interpretation. A review of 19 questionnaire-interview comparison studies found that consensus and consistency statistics were generally weak between…

When attempting to determine if an intervention has a causal impact, the "gold standard" of program evaluation is the randomized controlled trial (RCT). In education studies random assignment is rarely feasible at the student level, making RCTs harder to conduct. School-level assignment is more common but this often requires considerable resources…

This paper discusses how a seldom-used statistical procedure, recursive regression (RR), can numerically and graphically illustrate data-driven nonlinear relationships and interaction of variables. This routine falls into the family of exploratory techniques, yet a few interesting features make it a valuable compliment to factor analysis and…

The paper presents a rubric to help evaluate the quality of research projects. The rubric was applied in a competition across a variety of disciplines during a two-day research symposium at one institution in the southwest region of the United States of America. It was collaboratively designed by a faculty committee at the institution and was…

Studied perceptions and recommendations from 62 survey researchers about the use of electronic surveys. Reduction of costs, usefulness for prenotification and followup, and comparability with existing software were the most positive aspects cited. Limitations identified included sampling problems and confidentiality and privacy problems. (SLD)

Discusses access to and management and interpretation of online data. Considers the new ethical considerations arising when it is not clear whether participants in a conversation know or remember that they are being monitored. Also discusses data collection, preparation, preservation, and analysis. (SLD)

Discusses issues involved in data transformation to aid researchers who do not have extensive mathematical backgrounds. Focuses on the use of data transformation for normalization of variables. Emphasizes that researchers should understand their data before performing analyses. (SLD)

This document consists of papers published in the electronic journal "Practical Assessment, Research & Evaluation" during 2000-2001: (1) "Advantages of Hierarchical Linear Modeling" (Jason W. Osborne); (2) "Prediction in Multiple Regression" (Jason W. Osborne); (3) Scoring Rubrics: What, When, and How?" (Barbara M. Moskal); (4) "Organizational…

Suggests the routine use of replications in field studies, pointing out that it is usually possible to synthesize replications quantitatively using meta-analysis. Makes the c ase that this is especially attractive for investigators whose research paradigm choices are limited in the field environment. (SLD)

Provides an overview of content analysis, a powerful data reduction technique. It is a systematic, replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding. The technique extends far beyond simple word frequency counts, but requires careful definition of categories. (SLD)

Discusses the use of Web-based surveys and identifies several factors that influence data quality. Suggests that Web-based surveying, although very attractive, should be used with caution at present because of coverage bias or bias if sampled people do not have access to the Internet. (SLD)

Provides an overview of the concept of alignment and the role it plays in assessment and accountability systems. Discusses some methodological issues affecting the study of alignment and explores the relationship between alignment and test score interpretation. Alignment is not only a methodological requirement but also an ethical requirement.…

Introduces the problem of hierarchical, or nested, data structures and discusses how this problem is dealt with effectively. Provides examples of the pitfalls of not doing appropriate analyses. New software is making hierarchical linear modeling more user-friendly and accessible. (SLD)