Testing of directional dependence is a method to infer causal direction that recently has attracted some attention. Previous examples by e.g. von Eye and DeShon (2012a) and extensive simulation studies by Pornprasertmanit and Little (2012) have demonstrated that under specific assumptions, directional-dependence tests can recover the true causal direction between two variables. Simulation results are important in the evaluation of any statistical method, but they are necessarily less complex than real data that come with potential irregularities (e.g. departures from linearity, presence of confounders, etc.). In this article, we evaluate the performance of directional-dependence tests using benchmark data consisting of 65 variable pairs with known causal order. We find that between 21% and 43% of all cases are correctly classified using different directional-dependence tests that rely on differences in skew, kurtosis, or a combined measure. We then examine some of the assumptions of the directional-dependence test and find that for virtually all variable pairs, some assumptions are violated. When only pairs in which assumptions are fulfilled are selected, performance of all directional-dependence tests improves. We probe whether particular features of the variable pairs impact whether a test yields a correct or incorrect result, but find no strong predictors. Our findings provide a complimentary picture to previously conducted simulation studies, and highlight the fact that directional-dependence tests are prone to causal classification errors when key assumptions are violated. Such violations are potentially common in real data.