This article studies the ordinal reliability of (total) test scores. This study is based on a classical-type linear model of observed score (X), true score (T), and random error (E). Based on the idea of Kendall's tau-a coefficient, a measure of ordinal reliability for small-examinee populations is developed. This measure is extended to large (theoretically, infinite) examinee populations. This extended measure is identified as the population version of Kendall's tau-a. Appropriateness of the traditional product moment correlation-based measures, [TX] and .XX0, is reviewed. It is observed that [TX] and .XX0 are appropriate within the family of normal distributions of (T, E) but are inappropriate in the family of nonnormal distributions. A simulated data example is considered to show that the traditional product moment correlation-based measures may be misleading. Furthermore, practical procedures are suggested for estimating ordinal reliability. Real test score data are used for demonstration.