This article reports the lexical diversity of summaries written by experts and test takers in an empirical study and then interrogates the (in)congruity between the conceptualisations of "summary" and "summarize" in the literature of educational research and the operationalization of summarization tasks in three international English language tests in relation to their task directions and assessment criteria. These analyses show that summarization is not a uniform construct; rather, it is multidimensional in terms of its purposes, functions, and practices in real-life contexts, and as a consequence not particularly well operationalized in language tests and so in need of precise definitions for specific assessment contexts. This conclusion has implications for designing summarization tasks and assessing task performance. First, clear and transparent task directions are essential to ensure that test takers and evaluators share a common understanding of a test task; test takers need to be instructed what kind of summary they are expected to produce, particularly what information to include and to exclude. Second, because summarization may well be a unique type of writing process, it is important to employ parameters different from and additional to those for independent composition writing in order to measure the quality of a summary effectively. (Contains 4 tables and 3 footnotes.)