This paper reports on a classroom-based inquiry using quantitative methods conducted with Japanese EFL students' writing practice using ETS's Criterion. The purpose of the study is to examine the actual effects of teachers' feedback on students' writing on Criterion. Twelve university students in Japan participated in this study, while completing three Criterion writing assignments each. Six of the students received feedback from both a teacher and from the Criterion system, while six only received feedback from Criterion. The results of the study demonstrate that while the group who received feedback from a teacher showed improvement in some rhetorical features such as thesis statement and awareness of readers, the group that only received feedback from Criterion did not demonstrate major changes in those areas over their three assignments. From these results we can state that while Criterion can provide useful feedback to EFL learners, a teacher's feedback on early drafts of written work is essential for learners to be able to substantially improve their writing. Finally, some pedagogical implications concerning giving effective feedback while using Criterion will be discussed.