This article describes a research study that determined the depth of vocabulary knowledge of 28 intermediate ESL learners. The study was carried out with Bricklayer, a vocabulary assessment tool for L2 English which tested the ESL learners on 72 words. Two post-tests collected evidence for concurrent validity. A semantic distance test captured incremental knowledge for 36 words, but Bricklayer's predictive power for this partial knowledge was weak. A standard multiple-choice test of the remaining 36 words showed that Bricklayer predicted 61% of known words and 69% of unknown words; results were better for words which were "strongly" predicted to be known or unknown. These findings provide promise that Bricklayer's assessment paradigm assists in building up models of students' knowledge and behaviour in CALL [computer-assisted language learning] environments.