This study compared three common vocabulary test formats, the Yes/No test, the Vocabulary Knowledge Scale (VKS), and the Vocabulary Levels Test (VLT), as measures of vocabulary difficulty. Vocabulary difficulty was defined as the item difficulty estimated through Item Response Theory (IRT) analysis. Three tests were given to 165 Japanese students, resulting in five measures of vocabulary knowledge and four measures of word difficulty. Analyses included item and score factor analysis, unidimensionality, local independence, and correlations. Results indicate that these are reliable tests. Tests of unidimensionality suggest these tests are essentially measuring one major latent trait, which can be interpreted as a factor for word knowledge. Strong correlations of the scores with each other provide evidence of concurrent validity, and for the interpretation of the scores as indicative of word knowledge. Correlations with other methods of estimating word difficulty, such as transformed frequency, length of word, or number of syllables, suggest that of these methods, the log of frequencies from very large corpora gives the best estimate of word difficulty. However, direct testing of vocabulary difficulty appears to, in the words of Kreuz (1987), "provide a better account of recognition latencies than do counts based on printed word frequency" (p. 159).