ERIC Number: ED344947
Record Type: RIE
Publication Date: 1991-Dec
Reference Count: N/A
Least-Squares Models to Correct for Rater Effects in Performance Assessment.
Raymond, Mark R.; Viswesvaran, Chockalingam
This study illustrates the use of three least-squares models to control for rater effects in performance evaluation: (1) ordinary least squares (OLS); (2) weighted least squares (WLS); and (3) OLS subsequent to applying a logistic transformation to observed ratings (LOG-OLS). The three models were applied to ratings obtained from four administrations of an oral examination required for certification in a medical specialty. For any single administration, there were 40 raters and approximately 115 candidates, and each candidate was rated by four raters. R2 values for OLS and LOG-OLS models were comparable, while R2 values were substantially higher for the WLS model. The results indicate that raters exhibited significant amounts of leniency error, and that application of the least-squares models would change the pass-fail status of approximately seven percent to nine percent of the candidates. Ratings adjusted by the models demonstrated higher reliability and correlated slightly higher than observed ratings with the scores on a written examination. Six data tables are included. An appendix contains two additional tables presenting a sample rating matrix. There is a 41-item list of references. (Author/SLD)
Descriptors: Evaluators, Higher Education, Interrater Reliability, Least Squares Statistics, Licensing Examinations (Professions), Medical Students, Models, Performance Based Assessment, Physicians, Rating Scales, Scores, Testing Problems
American College Testing Research Report Series, P.O. Box 168, Iowa City, IA 52243.
Publication Type: Reports - Evaluative
Education Level: N/A
Authoring Institution: American Coll. Testing Program, Iowa City, IA.
Identifiers: Oral Examinations; Rater Effects; Rater Stringency Error