This study examined whether two variations on the typical Angoff group standard-setting process would produce sufficiently consistent results to recommend their use. The results imply that judgments gathered after an initial traditional group-process session can provide an efficient alternative mechanism for setting cutting scores using the Angoff…

This study investigated the effects of therapists' observer-client race pairing and client self-disclosure on observers' descriptive and attitudinal ratings of clients. A major implication is that observer race, client race, and client self-disclosure influence clinical decision-making. (Author/LHW)

Thirty-eight methods are presented for either setting standards or adjusting them based on an analysis of classification error rates. A trilevel classification scheme is used to categorize the methods, and 10 criteria of technical adequacy and practicability are proposed to evaluate them. (Author/LMO)

The validity of elementary school pupil ratings of the teaching performance of 33 student teachers was examined using the nine competencies of the Teacher Performance Assessment Instruments. Ratings of college supervisors and supervising teachers were criteria for contrast of validity coefficients of student ratings. (Author/BS)

This paper charts a cognitive history of the concepts of quantity and quality from three inter-related and inter-dependent perspectives of mathematics, logic, and physics. In so doing, other notions associated with the evolution of these concepts are identified and explicated. It is argued that the concepts of quantity and quality, considered in…

A new standard setting approach is introduced, called the cognitive components approach. Like the Angoff method, the cognitive components method generates minimum pass levels (MPLs) for each item. In both approaches, the item MPLs are summed for each judge, then averaged across judges to yield the standard. In the cognitive components approach,…

This paper discusses recent developments in language testing. It begins with a review of the traditional criteria that are applied to all measurement and outlines recent emphases that derive from the expanding range of stakeholders. Drawing on Alderson's seminal work, criteria are presented for evaluating communicative language tests. Developments…

Suggestions are provided for future research in item bias detection, reduction of essay-reader variation in setting cut-score levels, and limitations of equating theory. (TJH)

Differences in the Texas Teacher Appraisal System scores of teacher subgroups over 2 years were examined for 2,366 teachers for scores on individual domains, sums of scores of the 1st 4 domains, and overall summary performance scores, as well as appraiser differences. Implications for teacher evaluation are discussed. (SLD)

Whether adequate levels of interrater reliability could be obtained on a national, standardized examination using one examiner per observation was studied with 101 paired candidate observations on an examination for optometry. Results indicate that psychometrically sound judgments can be obtained with one examiner. (SLD)

Multifaceted Rasch analysis was used to determine whether bias was evident in the way a group of raters graded two different versions of an oral interaction test, undertaken by the same candidates. Results indicate that certain raters consistently rated the tape version of the test more harshly while others rated the live one more harshly. (10…

Issues concerning the selection of judges for standard setting are discussed. Determining the consistency of judges' recommendations, or their congruity with other expert recommendations, would help in selection. Enough judges must be chosen to allow estimation of recommendations by an entire population of judges. (SLD)

Training judges to generate item ratings in standard setting once the reference group has been defined is discussed. It is proposed that sensitivity to the factors that determine difficulty can be improved through training. Three criteria for determining when training is sufficient are offered. (SLD)

Medical school admission interview ratings for four entering classes (n=356 students) were compared with preadmission academic variables (admission test scores, undergraduate grades), student characteristics (age, gender, residence), and interviewer characteristics (gender, professional background, admission committee membership). Recommendations…

A study investigated the consistency of faculty and peer evaluations of the oral communication skills of 44 fourth-year pharmacology students. Substantial agreement between faculty and students was found. Peer evaluations were independent of their own communication skills. In addition, a significant correlation between oral and written…