Reviewing the test specifications to improve the quality of language tests may be a routine process in professional testing systems. However, there is a paucity of research about the effect of specifications review on improving the quality of small-scale tests. The purpose of the present study was twofold: how specifications review could help improve the validity of a test in the context of assessment for learning (AFL) and to what extent qualitative review of items can identify poor ones. To this end, a group of trained test specifications reviewers (No = 7) provided feedback on the specifications of a test and the quality of the items. Analysis of feedback showed that pedagogical concerns naturally emerged during the specifications review and led to concrete suggestions on how the specifications could be revised so that the resulting test could become more useful in a classroom context. Moreover, the test items were administered to a group of (No = 40) test-takers and a set of quantitative item analyses was carried out. Comparison of the results of qualitative judgment of reviewers with the quantitative analyses showed about 38% of the items suggested for revision by the reviewers were also identified as poor by the classical test theory (CTT) analysis. The findings highlight the potential of specifications review as part of the a priori validation of tests in small-scale assessments where conducting statistical analysis is not usually feasible.