Rating scales are a typical method for evaluating a student's performance in outcomes assessment. The analysis of the quality of information from rating scales poses special measurement problems when researchers work with faculty in their development. Generalizability measurement theory offers a set of techniques for estimating errors or unreliability due to a variety of errors. In this study, generalizability calculations were made for two sets of data, one from the arts and one from the sciences. Followup decision studies were calculated to determine the adequacy of the number of raters (interrater reliability) and the number of rating items. The rationale, terminology, and implications of generalizability theory are discussed for institutional researchers. Contains 7 references. (Author/KM)