Language and culture endow humans with access to conceptual information that far exceeds any which could be accessed by a non-human animal. Yet, it is possible that, even without language or specific experiences, non-human animals represent and infer some aspects of similarity relations between objects in the same way as humans. Here, we show that monkeys' discrimination sensitivity when identifying images of animals is predicted by established measures of semantic similarity derived from human conceptual judgments. We used metrics from computer vision and computational neuroscience to show that monkeys' and humans' performance cannot be explained by low-level visual similarity alone. The results demonstrate that at least some of the underlying structure of object representations in humans is shared with non-human primates, at an abstract level that extends beyond low-level visual similarity. Because the monkeys had no experience with the objects we tested, the results suggest that monkeys and humans share a primitive representation of object similarity that is independent of formal knowledge and cultural experience, and likely derived from common evolutionary constraints on object representation.