In this study, a hierarchical linear model (HLM) was used within a cluster random assignment design to test the impact of "Criterion"? on students' writing skill when "Criterion" was used as a supplementary instructional tool in ninth grade English classrooms over a 4 week period. The study was conducted in three states: Indiana, Pennsylvania, and New Jersey. The sample consisted of 1,464 ninth graders aggregated across 10 schools, 34 teachers, and 61 classes. Findings indicate that over a short time period, "Criterion" use improved the mechanical aspects of student essays as measured by the analytic scores given to their essays. Students in ninth grade English classes who used "Criterion" several times a week over a 1 month period received higher analytic scores on their essays written at the end of the study period than those who were in classes that did not. The effect was small, but statistically significant at the 0.05 level using a one-tailed test. These findings suggest a role for automated feedback systems in supporting writing instruction.