ERIC Number: ED173445
Record Type: RIE
Publication Date: 1979-Apr
Reference Count: 0
The Reliability of Auditor Judgments: A Meta-evaluation Study.
Bergquist, Constance C.
The reliability of evaluators' judgments was investigated, using a discrepancy evaluation model within a program evaluation context. Two research problems were addressed: the degree of evaluator reliability in detecting and rating problems in federal educational project planning, management, and evaluation; and the relationship between inter-judge reliability and the types of items examined, the evaluators' expertise, and the type of project being evaluated. Of thirty-one evaluators studied, 77% had doctoral degrees, and all had previous training related to the Title IV-C audit system. Three types of reliability coefficients were calculated: (1) the consistency of a single rater over a set of ratings was .07 to .64; (2) the estimated single rater reliability over all situational protocols was .20 to .84, with most coefficients above .50; and (3) the average interrater reliability was .45 to .96, with over half of the coefficients above .80. Results raised two questions about the reliability of judgments made by educational evaluators; questionable replicability of evaluation methods and standards, and the amount of disagreement among evaluators on methods and procedures for evaluating programs. (MH)
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers: Auditors; Audits; Florida; Interrater Reliability
Note: Paper presented at the Annual Meeting of the American Educational Research Association (63rd, San Francisco, California, April 8-12, 1979)