The purpose of this study was to compare the ability of different instruments, independently developed and traditionally used for measuring science teachers' beliefs in short-term interventions, to longitudinally measure teachers' changing beliefs. We compared the ability of three self-report instruments (Science Teaching Efficacy Belief Instrument Form A [STEBI], Teaching of Science as Inquiry instrument [TSI], Inquiry Teaching Beliefs instrument [ITB]) and one observational instrument (Reformed Teaching Observation Protocol [RTOP]) to appropriately measure high school chemistry teachers' beliefs as they engaged in a two-and-a-half year professional development program. Collectively our findings from these four instruments, across three separate cohort of teachers (N = 16), indicated conflicting changes in teacher beliefs. For example, the STEBI indicated teachers' self-efficacy remained unchanged or increased while the TSI indicated a concurrent decrease in self-efficacy throughout the PD program. Additionally, the ITB seemed to indicate a decrease in teachers' knowledge of inquiry while their interview data and RTOP scores indicated a concurrent increase in their knowledge of and ability to enact inquiry-based practices. We reconcile these conflicting results and discuss the implications these findings have for validly and reliably measuring science teacher belief changes within longer duration PD.