Collaborative problem solving is a complex skill set that draws on social and cognitive factors. The construct remains in its infancy due to lack of empirical evidence that can be drawn upon for validation. The differences and similarities between two large-scale initiatives that reflect this state of the art, in terms of underlying assumptions about the construct and approach to task development, are outlined. The goal is to clarify how definitions of the nature of the construct impact the approach to design of assessment tasks. Illustrations of two different approaches to the development of a task designed to elicit behaviors that manifest the construct are presented. The method highlights the degree to which these approaches might constrain a comprehensive assessment of the construct.