Comparisons were made among various methods of estimating the reliability of pass-fail decisions based on mastery tests. The reliability indices that are considered are p, the proportion of agreements between two estimates, and kappa, the proportion of agreements corrected for chance. Estimates of these two indices were made on the basis of repeated or alternate test administrations and also on the basis of a single administration with the assumption of a beta-binomial or negative-hypergeometric distribution. Across a variety of test score distributions, test lengths, and mastery (cutoff) scores, the beta-binomial estimates tended to underestimate the corresponding population values. The percent of bias was small (about 2.5%) for p and somewhat larger (about 10%) for kappa. Both beta-binomial estimates had standard errors about one-half the size of the standard errors of estimates based on repeated test administrations. Though the beta-binomial estimates presumed equality of item difficulty, the data presented indicated that even gross departures from equality of item difficulty did not affect the amount of bias of the estimates. (Author/CTM)