ERIC Number: ED336429
Record Type: RIE
Publication Date: 1990-Dec
Reference Count: N/A
Detecting and Correcting for Rater Effects in Performance Assessment.
Raymond, Mark R.; Houston, Walter M.
Performance rating systems frequently use multiple raters in order to improve the reliability of ratings. However, unless all candidates are rated by the same raters, some candidates will be at an unfair advantage or disadvantage solely because they were rated by more stringent or lenient raters. To obtain fair and accurate evaluations of candidate performance, such sources of systematic rating error must be considered. This paper describes four procedures to detect and correct for rater effects: (1) ordinary least squares; (2) weighted least squares; (3) the Rasch model; and (4) data imputation via the E-M algorithm. A demonstration of each procedure is provided, using a small set of data for 25 individuals rated by six raters, to illustrate the procedures. Data were simulated to provide the types of ratings the might be obtained from performance evaluations conducted in settings such as military training schools; physician residency programs; and other work settings in business, industry, or government. Demonstration results, which are consistent with those of other research studies, indicate that each of the methods produces more accurate estimates of true levels of performance than the traditional approach of summing observed ratings. Six tables present data from the demonstrations. A 40-item list of references is included. (Author/SLD)
Descriptors: Algorithms, Computer Simulation, Educational Assessment, Evaluation Methods, Evaluators, Interrater Reliability, Least Squares Statistics, Occupational Tests, Performance, Performance Based Assessment, Personnel Evaluation, Rating Scales, Scoring, Test Bias, Test Reliability
ACT Research Report Series, P.O. Box 168, Iowa City, IA 52243.
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: American Coll. Testing Program, Iowa City, IA.
Identifiers: EM Algorithm; Rasch Model; Rater Effects
Note: An earlier version of this paper was presented at the Annual Meetings of the American Educational Research Association (Boston, MA, April 16-20, 1990) and the National Council on Measurement in Education (Boston, MA, April 17-19, 1990).