For over a decade, the authors have been involved in a design theory experiment providing software for high school students preparing for the descriptive question on the Advanced Placement (AP) chemistry examination. Since 1997, the software has been available as a Web site offering repeatable practice. This study describes a 4-year project during which incorrect responses using the most recent interface were collected and saved in a dataset. A descriptive analysis suggests that the most basic errors include: failing to write appropriate chemical formulas, recognize reactive species in net ionic reactions, and recognize weak electrolytes. The largest single group of reported errors, recognizing weak electrolytes, implies strategies for improved student performance on the AP examination. In addition, the results of this study support the development of greatly enhanced feedback for future learners who use the site. (Contains 3 tables.)