One of the most fundamental aspects of learning a language is determining the mappings between words and referents. An often-overlooked complication is that infants interact with multiple individuals who may not produce words in the same way. In the present study, we explored whether 10- to 12-month-olds can use talker-specific knowledge to infer the intended referents of novel labels. During exposure, infants heard two talkers whose front vowels differed; one talker trained them on a word-referent mapping. At test, infants saw the trained object and a novel object; they heard a single novel label from both talkers. When the label had a front vowel (Experiment 1), infants responded differently as a function of talker, but when it had a back vowel (Experiment 2), they did not, mapping the novel label to the novel object for both talkers. These results suggest that infants can track the phonetic properties of two simultaneously presented talkers and use information about each talker's previous productions to guide their referential interpretations.