This study investigates the consistency between human raters and an automated essay scoring system in grading high school students' English compositions. A total of 923 essays from 23 classes of 12 senior high schools in Taiwan (Republic of China) were obtained and scored manually and electronically. The results show that the consistency between human raters is significantly higher than the consistency between human raters and the automated essay scoring system. To discover whether the students were confident with the automated essay grading system, a questionnaire was also distributed. The results indicate the participants realize their vocabulary level was inadequate, and they wanted to know the scores the automated essay grading system gave them along with the generated comments regarding their compositions. (Contains 6 tables.)