Reliability is a problem inherent in all educational assessments, but the amount of attention this particular problem should be given is related to the function and use of the assessment. In this article, classification accuracy is put forward as a conceptualization of reliability that is meaningful for a large number of educational assessments. Furthermore, a fairly transparent method of estimating the magnitude of classification accuracy is developed. The article examines how the classification accuracy of a Swedish national test in mathematics is affected by changes in the test that are motivated by other validity concerns. The inquiry is based on the results on 1201 students participating in a Swedish national test in mathematics. The results indicate that there can be a significant trade-off between reliability and other aspects of validity.