This article critically examines the methodology of the Shanghai Academic Ranking of World Universities (ARWU) by generating raw scores for the "big five" South African research universities (Stellenbosch, Cape Town, Kwazulu-Natal, Pretoria and the Witwatersrand, henceforth referred to as SU, UCT, UKZN, UP and WITS) using the ARWU indicators. The performance of SU (which has never been ranked among the top 500 universities) is compared to that of the other four universities. The difference between SU and UKZN is that UKZN has one highly cited scientist and four more "reprint author" affiliations in the journals Nature and Science. UP lost its ranking in the 2008 ARWU but its performance does not show any significant change in terms of the number of articles on which the PUB and N & S indicators are based. Some ARWU indicator scores violate the original proportionality in the source data. The notion of whether a recipe exists for achieving a ranking is discussed, together with the possible consequences of steering institutional research practices towards meeting the ARWU criteria. (Contains 11 tables.)