Distractor-driven multiple-choice assessment items and Rasch modeling were used as diagnostic tools to investigate students' understanding of middle school chemistry ideas. Ninety-one items were developed according to a procedure that ensured content alignment to the targeted standards and construct validity. The items were administered to 13360 middle school, high school, and college students from across the USA, and the student data were analyzed using Rasch modeling. A cross-sectional analysis was performed to examine the progression of understanding of chemistry from middle school to college and revealed an overall increase in understanding with increasing grade level. Option probability curves for several of the items were used to provide insight into how students' thinking changes with increasing knowledge of the ideas being tested. In some cases, hierarchies of misconceptions were detected in which the misconceptions decrease in a series as the overall student performance level increases. Additionally, for one item, the peculiar shape of the option probability curve for a distractor indicated a flaw in the item. (Contains 2 tables and 8 figures.)