The paper tackles a central question in the field of Intelligent Computer-Assisted Language Learning (ICALL): How can language learning tasks be conceptualized and made explicit in a way that supports the pedagogical goals of current Foreign Language Teaching and Learning and at the same time provides an explicit characterization of the Natural Language Processing (NLP) requirements for providing feedback to learners completing those tasks? We argue that the successful implementation of language learning tasks that can be automatically assessed demands a design process in which both pedagogical and computational requirements feed each other. Extending well-established work in Task-Based Instruction (TBI) and Language Testing, we propose a framework that helps us (a) elucidate the formal features of foreign language learning activities, (b) characterize the relation between the expected and the actually elicited learner language, and (c) assess how the variability in the learner responses impacts interpretation and computational techniques for the automatic analysis of learner language. To validate our approach we apply the framework to two writing tasks for learners of English as a foreign language. Our analysis highlights the relevance of spelling out the pedagogical and linguistic goals of language learning tasks in order to successfully characterize the language expected and actually elicited in learner responses, to evaluate the realization of the task, and to support the design of effective NLP strategies. Given the combination of design- and data-driven perspectives, the framework supports an iterative approach to the creation of language learning tasks and ICALL materials.