In their first year, infants begin to learn the speech sounds of their language. This process is typically modeled as an unsupervised clustering problem in which phonetically similar speech-sound tokens are grouped into phonetic categories by infants using their domain-general inference abilities. We argue here that maternal speech is too phonetically variable for this account to be plausible, and we provide phonetic evidence from Spanish showing that infant-directed Spanish vowels are more readily clustered over word types than over vowel tokens. The results suggest that infants' early adaptation to native-language phonetics depends on their word-form lexicon, implicating a much wider range of potential sources of influence on infants' developmental trajectories in language learning.