The reliability of evaluators' judgments was investigated, using a discrepancy evaluation model within a program evaluation context. Two research problems were addressed: the degree of evaluator reliability in detecting and rating problems in federal educational project planning, management, and evaluation; and the relationship between inter-judge reliability and the types of items examined, the evaluators' expertise, and the type of project being evaluated. Of thirty-one evaluators studied, 77% had doctoral degrees, and all had previous training related to the Title IV-C audit system. Three types of reliability coefficients were calculated: (1) the consistency of a single rater over a set of ratings was .07 to .64; (2) the estimated single rater reliability over all situational protocols was .20 to .84, with most coefficients above .50; and (3) the average interrater reliability was .45 to .96, with over half of the coefficients above .80. Results raised two questions about the reliability of judgments made by educational evaluators; questionable replicability of evaluation methods and standards, and the amount of disagreement among evaluators on methods and procedures for evaluating programs. (MH)