The purpose of mathematics competitions, and in our case the South African Mathematics Olympiad (SAMO), is to promote problem solving skills and strategies, to generate interest and enthusiasm for mathematics and to identify the most talented mathematical minds. SAMO is organised in two divisions--a junior and a senior division--over three rounds. We analysed the results of the junior second round over seven years, 2006-2012. Based on the literature, a mathematical content framework was developed, dividing the mathematical content into seven broad content areas. In this paper we investigate the face validity, diagnostic attributes and predictive criterion validity of mathematics olympiad question papers over the period by focussing on the frequency of content area occurrence in the different items. We also look at the performance of contestants in the different content areas as a broad diagnosis. Lastly we investigate the item performance, comparing the expected performance by the problem committee of experts setting the question papers and the actual performance of contestants. Topics such as numbers, algebra, patterns and functions, measurement, applications, modelling and logic were used abundantly whereas (quite surprisingly) there were few items on graphs, decimal fractions, spatial logic and vertices and edges of polygons, indicating that the face validity can be improved. Contestants performed best in items on algebra and weakest in items on statistics. The ability of the problem committee to anticipate student item performance varied considerably and was significantly worse in 2012 than in 2006, indicating better predictive criterion validity in 2006.