Online vocabulary learning tools have been gaining popularity in EFL classrooms. Although a considerable number of research papers have showed its effectiveness, the amount of research on learner progress in terms of productive skills is scarce. Little is also known about what types of learners make progress after using these tools. This is largely due to a lack of consideration of learning strategies and proficiency levels. The present study investigates the effectiveness of an online vocabulary learning device in Japan on the gains in passive and active vocabulary use at a range of proficiency levels. A total of 58 Japanese university students took a pre-instruction test and two post-instruction tests including vocabulary, reading, and essay-writing, allowing the author to investigate the degree of learner progress in mastery of passive and active vocabulary use. Additionally, they completed pre- and post-instruction questionnaires to confirm the device's impact on their progress. The results emerging from cluster analysis and ANOVA verified the use of the device for short-term retention of passive vocabulary for intermediate and advanced learners, but there seemed to be little improvement in active vocabulary use among learners at all proficiency levels. The study highlights the need to include the tool's functions that enhance student levels of word production as well as long-term retention.