Over two semesters short essay prompts were developed for use with the Graphical Interface for Knowledge Structure (GIKS), an automated essay scoring system. Participants were students in an undergraduate-level online introductory statistics course. The GIKS compares students' writing samples with an expert's to produce keyword occurrence and links in common scores which can be used to construct a visual representation of an individual's knowledge structure. Each semester, students responded to the same two essay prompts during the first and last week of the course. All responses were scored by the GIKS and two instructors. Evidence for the validity of scores obtained using the GIKS was provided through the use of correlations with instructors' scores and final exam scores. Changes in scores from the beginning to end of the course were examined. Suggestions for writing open-ended prompts that work well with computer-automated scoring systems are given as well as suggestions for using the GIKS as a formative learning activity as opposed to summative assessment.