We investigated the impact of perceptual and categorical relatedness between a target and a distracter object on early referent identification in infants and adults. In an intermodal preferential looking (IPL) task, participants looked at a target object paired with a distracter object that could be perceptually similar or dissimilar and drawn from the same or different global category. The proportion of target looking measures revealed that infants and adults were sensitive to the interplay between category membership and perceptual similarity. Online latency measures demonstrated an advantage for perceptually dissimilar items regardless of their categorical status, indicating that different IPL measures index different processes during target identification. Results suggest that perceptual similarity and category membership of the objects lead to competition effects in word recognition and referent identification in both adults and infants and that lexical categorization and nonlinguistic categorization processes are closely related during infancy. (Contains 3 figures and 3 tables.)