There are two main lines of research in estimating classification accuracy (CA) and classification consistency (CC) under Item Response Theory (IRT). The R package cacIRT provides computer implementations of both approaches in an accessible and unified framework. Even with available implementations, there remains decisions a researcher faces when choosing and applying the best approach for the situation. This paper identifies and discusses the practical issues that researchers may face when estimating CA and CC. To exemplify the analytic decisions, both approaches are applied to a common dataset with discussion. In addition to generalizable guidance, the demonstration provides R code for the cacIRT package.