ERIC Number: ED174680
Record Type: RIE
Publication Date: 1979-Apr
Reference Count: 0
Estimating the Reliability of Clinical Ratings: An Alternative Approach.
Smith, Douglas U.
The ratings of 43 third-year medical students were used to investigate a method of computing the reliability of ratings. The students were evaluated on a four-point rating scale along seven dimensions of clinical ability: diagnostic ability; motivation; sense of responsibility; oral-verbal ability; effectiveness with patients; ability to take a history and perform a physical examination; and cooperation. Seven individuals rated these students. The ratings were analyzed by means of a computer program which used a pairwise definition of agreement, and provided the following information: (1) distribution of ratings for each rater; (2) number of agreements as well as the proportion of times the rater was in agreement with other raters with whom he was paired; (3) kappa coefficient (corrected for chance agreement) for each rater on each dimension; (4) maximum value of kappa given the marginal distributions of ratings; and (5) mean and standard deviation of ratings. This information was also summarized for each rater across all dimensions, as well as for all raters on each dimension. An analysis of variance estimate of reliability was computed for comparison. The author's method provided greater meanings for the estimates, as well as an indication of the sources of unreliability. (Author/GDC)
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers: Interrater Reliability; Kappa Coefficient; Rater Reliability; Reliability Formulas
Note: Paper presented at the Annual Meeting of the American Educational Research Association (63rd, San Francisco, California, April 8-12, 1979)