The aim of this study was to compare the results of many-facet Rasch analyses based on crossed and judge pair designs. The study was conducted with 168 eighth grade students and five judges. The study data were collected using an achievement test with open-ended questions and a holistic rubric that was used to rate the responses. In the data collection process, the achievement test was initially administered to students. The students' responses were rated by five judges; the ratings provided the data set for the crossed design. In the judge pair design, the students' responses to the test items were rated by any two of the five judges. Then the researcher conducted the many-facet Rasch analysis and compared the analysis outputs of the two designs. It was determined that the difference between the elements of the facets included in the analysis was demonstrated more effectively in the crossed design than the judge pair design. In the judge pair design, however, the study made significant distinctions between the elements of the facets. Moreover, the study found that the relative agreement between the ability estimations calculated based on crossed and judge pair designs was very high, and there was no significant difference between the ability estimations reported by the two designs. Therefore, it is preferable to choose judge pair design for examinations with a large number of participants as it will be less costly.