Simple arithmetic word problems are often featured in elementary school education. One type of problem, "compare with unknown reference set," ranks among the most difficult to solve. Differences in item difficulty for compare problems with unknown reference set are observed depending on the direction of the relational statement (more than vs. less than). Various cognitive models have been proposed to account for these differences. We employed item response theory (IRT) to compare competing cognitive models of student performance. The responses of 100 second-grade students to a series of compare problems with unknown reference set, along with other measures of individual differences, were fit to IRT models. Results indicated that the construction integration model (Kintsch, 1988, 1998) provided the best fit to the data. We discuss the potential contribution of psychometric approaches to the study of thinking.