Viewing the reliability for criterion-referenced tests as that of mastery classification decisions, three models for determining reliability were examined using two test administrations so that two estimates could be compared to a standard. A major purpose of the research was to determine how several reliability coefficients (coefficient kappa, an estimate of kappa, and a coefficient of agreement) were affected by test length, test length and cutoff score combined, student ability and cutoff score combined, sample size, and test content heterogeneity. The relationships between reliability and validity for criterion-referenced tests, and between norm-referenced and criterion-referenced reliability for the same data, were also examined using the Instructional System in Mathematics (ISM) program, an objectives-based instructional management system for grades K-8. A sample of 325 students were tested with three criterion-referenced tests before and after a two-week spring vacation. Coefficient kappa, which can only be calculated when there are two test administrations, generally had the lowest values and the largest standard errors. The estimate of coefficient kappa had slightly larger mean values and smaller standard errors. The coefficient of agreement was the highest reliability coefficient. (CM)