The relationship between the characteristics of 33 mathematical problem-solving questions answered by 16-year-old students in England and the quality of problem-solving elicited was investigated in two studies. The first study used comparative judgement (CJ) to estimate the quality of the problem-solving elicited by each question, involving 33 mathematics teachers judging pairs of journal-style responses to the questions and the application of the Bradley-Terry model. In the second study a variant of Kelly's Repertory Grid was used with five mathematics teachers to identify 23 dimensions along which the problem-solving questions varied. Significant relationships between ratings on some dimensions and the problem-solving quality estimated in the first study were found. This suggests that the Kelly's Repertory Grid approach could be an effective way to identify features of questions that are relevant to the construct being assessed and features that could be potential sources of construct-irrelevant variance in test scores.