As part of a project to translate and administer the Torrance Tests of Creative Thinking (TTCT) to Turkish elementary and secondary students, 35 professionals were trained in a full-day workshop to learn to score the verbal TTCT. All trainees scored the same 4 sets of TTCT verbal criterion tests for fluency, flexibility, and originality by filling out specially designed forms to allow item analysis of each of the criterion responses. This analysis outlines a protocol to monitor the responses of scorers at the item level, not a level of analysis that is normally performed on the TTCT. The results of item level analyses for fluency and flexibility data from the 35 trainees reveal new information about translation, training, and scoring. Based on these results, this article makes the case for item-level analysis for cross-cultural applications to prevent artificially congruent scores that will not predict accuracy in future scoring.