In recent decades, technology has influenced various aspects of assessment in mathematics education: (1) supporting the assessment of higher-order thinking skills in mathematics, (2) representing authentic problems from the world around us to use and apply mathematical knowledge and skills, and (3) making the delivery of tests and the analysis of results through psychometric analysis more sophisticated. We argue that these developments are not pushing mathematics education in the same direction, however, which creates tensions. Mathematics education--so essential for educating young people to be creative and problem solving agents in the twenty-first century--is at risk of focusing too much on assessment of lower order goals, such as the reproduction of procedural, calculation based, knowledge and skills. While there is an availability of an increasing amount of sophisticated technology, the related advances in measurement, creation and delivery of automated assessments of mathematics are however being based on sequences of atomised test items. In this article several aspects of the use of technology in the assessment of mathematics education are exemplified and discussed, including in relation to the aforementioned tension. A way forward is suggested by the introduction of a framework for the categorisation of mathematical problem situations with an increasing sophistication of representing the problem situation using various aspects of technology. The framework could be used to reflect on and discuss mathematical assessment tasks, especially in relation to twenty-first century skills.