ERIC Number: ED186474
Record Type: RIE
Publication Date: N/A
Reference Count: 0
Performance Test Objectivity: Comparison of Interrater Reliabilities of Three Observation Formats.
Nugent, William A.; Laabs, Gerald J.
Comparisons were made of three different procedures for rating job performance and the validity of each procedure was estimated. All three procedures were based on observation of videotaped presentations of 12 electrical measurement problems, 8 of which had errors associated with them. The three procedures involved the use of three different evaluation forms: structured, semi-structured, and unstructured. The unstructured and semi-structured forms were modeled after existing Navy forms; but the structured form, requiring the evaluation of a series of explicit procedural steps, was specially designed for this study. Of the 78 raters tested, 42 were considered proficient and 36 were classified as low skill proficient. The percent agreement of raters using the structured rating form with correct job performance was nearly perfect (97 percent), but the agreement for the semi-structured form was only 81 percent and for the unstructured form was 77 percent. The percent of rating errors of omission and commission was also observed as were differences between high skilled and low skilled raters. (CTM)
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers: Evaluation Problems; Interrater Reliability; Navy
Note: Paper presented at the Annual Meeting of the Military Testing Association (20th, Oklahoma City, OK, October 30-November 3, 1978).