Assessment instruments developed specifically for chemistry classrooms have increased in number over the past decade. In the design, development, and evaluation of these instruments, the chemical education community has adopted many of the practices and standards used by the greater assessment communities. Methodologies for creating new assessment instruments now include collecting broad evidence for the validity of the uses and interpretations of data derived from an assessment instrument. The focus of this study is the design, development, and qualitative evaluation of concept inventory items for the Thermochemistry Concept Inventory (TCI). Qualitative research studies were used to obtain feedback from the primary stakeholders of the TCI. Evidence for content and response process validity are provided and used as arguments against the two threats to validity: construct underrepresentation and construct-irrelevant variance. In addition, a determination of the most important thermochemical topics taught in general chemistry classrooms is derived using feedback from general chemistry instructors' responses to an online survey. Semistructured, think-aloud interviews were used to identify alternative conceptions used by students answering open-ended questions. Qualitative data were used to develop a series of multiple-choice items that were then evaluated by students in retrospective think-loud interviews. These student interviews, along with additional interviews with general chemistry instructors, provide evidence for the response process validity of the items.