This paper sets out the findings from a large-scale analysis of the Northern Ireland Transfer Procedure Tests, used to select pupils for grammar schools. As it was not possible to get completed test scripts from government agencies, over 3000 practice scripts were completed in simulated conditions and were analysed to establish whether the tests present a valid means of selection. The analysis replicated the official processes using age adjustments, standardization and weightings. The results, which have been officially conceded, are startling. The highest and lowest grades (A and D) were separated by as few as 18 of the total of 150 available marks and the standard error of measurement was of the order of 4.75. The most serious implication of this finding is that the candidate ranking system has the potential to misclassify up to two-thirds of the test-taking cohort by as many as three grades. The paper also highlights other major problems in the tests' design, for example, their 'easiness', which causes children with 70% of the answers correct to be awarded a D grade, and argues the need for compliance with international educational testing standards.