A 2011 report by the Department of Education states that understanding how teachers use results from formative assessments to guide their practice is necessary to improve instruction. Chemistry teachers have goals for items in their formative assessments, but the degree of alignment between what is assessed by these items and the teachers' goals has not previously been investigated. This understanding of teachers' goal-setting will identify strengths and limitations in their formative assessment processes. In this qualitative project, we have characterized this alignment of assessment items with learning objectives with data collected from 19 high school chemistry teachers from 10 states. These teachers participated in semi-structured interviews describing their goals and use of a teacher-developed formative assessment they had administered to their classes. The teachers provided 41 items which were analyzed for this study. To evaluate the content knowledge and skills required for successful completion of each item, the items were evaluated by one researcher by listing statements and equations required to demonstrate an understanding and solve each individual problem. A different member of the team who conducted the interviews listed the assessment goals as stated by the teachers for each item to characterize the degree teacher's goals are emulated by their items. The author team evaluated the alignment of teachers' assessment goals to what is assessable by items. The results discussed show that teachers who address conceptual chemical phenomena in their goals, in line with the NGSS standards of using concepts to assess student understanding, are more likely to have a goal that is assessable by their assessment items than those who do not assess concepts. From findings and prior literature, we recommend that teachers use conceptual goals in conjunction with problem solving goals to assess student understanding, particularly with items that require computation.