The Framework for K-12 Science Education (NRC, 2011) outlines core disciplinary ideas, scientific practices and cross-cutting ideas as dimensions on which to base science education. This study outlines the use of core ecological ideas and two scientific practices as a way to examine the cognitive complexity of released large-scale assessment items in Ecology. We present results from analysis of 212 released items from state, national and international sources and compare the types of core ecological ideas and scientific practices required to successfully complete these assessment tasks. We administered a subset of items to students to examine how well our rubric was aligned with students' interactions with the items. Our findings indicate that different sources of items have different proportions of item formats and have items designed to elicit different types of core ideas and practices from students. The analysis of students' responses to items indicates that using a framework of core ecological ideas and scientific practices provide a good indication of item difficulty. However, this paper highlights several instances where the coding of items does not match students' responses to items and we analyze possible reasons for the discrepancies -- including both item design flaws and opportunity to learn.