NotesFAQContact Us
Search Tips
ERIC Number: ED318776
Record Type: Non-Journal
Publication Date: 1990-Apr
Pages: 28
Abstractor: N/A
Reference Count: N/A
Rater Stringency and Consistency in Performance Assessment.
Webb, Lynn C.; And Others
Two aspects of rater accuracy in performance assessment were analyzed: rater stringency/leniency, and rater consistency. Data were obtained from three administrations of an oral certification examination in a health profession. The examination consists of clinical cases in four content areas or subspecialities. A total of 364 candidates were examined by a total of 60 raters over the 3-year period (1987-89) of the study. In any 1 year, approximately 120 candidates are examined, each by 4 raters; each rater examined from 9 to 14 candidates. Candidates were given three scores by each rater; the scores are on a Likert-type rating scale ranging from 1 to 12. Rater bias was studied using an ordinary least squares approach. Rater consistency was studied by computing each rater's mean-squared deviation from the model. Rater bias could be easily corrected using a simple regression model. Rater inconsistency, when discovered, presented a more complex problem. Inconsistency may be best addressed during the rater training phase. Eight tables present study data. (SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A