This article presents a study of EssayCritic, a computer-based writing aid for English as a foreign language (EFL) that provides feedback on the content of English essays. We compared two feedback conditions: automated feedback from EssayCritic (target class) and feedback from collaborating peers (comparison class). We used a mixed methods approach to collect and analyze the data, combining interaction analysis of classroom conversations during the writing process and statistical analysis of students' grades. The grades of students in both classes improved from pre-test to post-test but in different ways. The students in the target class included more ideas (content) in their essays, whereas the students in the comparison class put more emphasis on the organization of their ideas. We discuss our findings to identify strengths and weaknesses of our approach, and we end the paper by suggesting some directions for further research.