In more and more countries, the goal of mathematics education is to develop students' mathematical competence beyond procedural and conceptual knowledge. Thus, students need to engage in a rich variety of tasks comprising competencies such as communication, reasoning and problem-solving. In addition, assessment tasks should be constructed to measure the particular aspects of mathematical competence to align assessment with curriculum. Teachers need to be able to recognise competency demand in the mathematical tasks they want to use for teaching and assessment purposes, which might prove difficult. The aim of this study was to analyse the outcome of 5 teachers' and prospective teachers' use of an item analysis tool. For each of 141 assessment tasks, the teachers and prospective teachers individually applied the tool to identify the competency demand of the task on a scale from 0 to 3 for each of 6 mathematical competencies. Overall, the analysis reveals high consistency in their analysis. However, the teachers and prospective teachers utilised a restricted range of the scale, rarely judging a task to demand a high level of competence. This indicates that the 5 teachers and prospective teachers can use the tool to identify which of the 6 competencies are at play in solving a task, but can only differentiate to a limited extent between tasks that demand a low level of competence and those that demand a high level. In conclusion, we propose that an item analysis tool could be useful to teachers and prospective teachers as a means of analysing and selecting appropriate tasks that enhance development of mathematical competencies.