An underlying goal in most chemistry curricula is to enable students to think like chemists, yet there is much evidence to suggest that students can learn to solve problems without thinking conceptually like a chemist. There are few tools, however, that assess whether students are learning to think like Ph.D. faculty, putative experts in the field. Here, we present a card-sorting task that probes how individuals organize information about problems in chemistry. Chemistry faculty tend to organize around "deep" features centered on fundamental ideas in chemistry while novices tend to organize around "surface" features such as problem presentation or specific vocabulary. We used established statistical techniques from card-sorting tasks in other fields and introduce a new quantitative measure that compares individual performance on the sorting task to faculty and novices that is hypothesis-independent. Initial results indicate that the card-sorting task is effective at distinguishing between populations of faculty and novices in chemistry and can be used to track progress toward more expert-like thinking over time through a chemistry education program.