In this article we describe the challenges associated with assessing complex competencies envisioned as the targets of learning in the 21st century. Such competencies typically reflect the integration of multiple dimensions of knowledge and skill. Technology plays a crucial role in their assessment, from conceptualisation through design, data gathering and interpretation of results. We use the case of science proficiency to illustrate challenges associated with the assessment of the intended products of multidimensional learning and the benefits provided by technology. We frame assessment development as an evidence-centered design process and illustrate it by using cases drawn from middle school science. We then turn to ways in which assessment systems need to evolve to expand the scope of what can be done in the creation and use of valid, reliable and equitable assessments of complex, multidimensional learning. We conclude by discussing policy implications of technology-based assessment systems with an emphasis on "measuring what matters" versus "measuring what is easy," since what we choose to assess will become the focus of instruction. Major advances in assessment policy and practice require investment in the development, validation and deployment of technology-based assessments that reflect the multidimensional competencies identified by contemporary research and theory.