Metabolic systems form the very foundation of life and as such are broadly taught in the molecular life sciences. Here, we describe the biochemistry educator community-based development and use of an assessment instrument designed to evaluate students' ideas about metabolic pathway dynamics and regulation in undergraduate biochemistry courses. Analysis of student responses showed that most students were able to interpret visual representations in an unfamiliar metabolic pathway and that many could make basic predictions about how the system would be expected to respond to changes. However, fewer students generated nuanced responses that accounted for both microscopic changes at the protein level and macroscopic changes in pathway product outputs. These findings identify some of the challenges of meaningfully assessing students' understanding of metabolic pathways and could inform how instructors think about teaching and assessing metabolism in undergraduate biochemistry and beyond. The results also suggest future avenues for biochemistry education research.