This research investigated how children build up the language-specific system of the color lexicon, examining factors that play important roles for the construction of an adult-like color lexicon. We had 3-, 4-, and 5-year-old Japanese-speaking children and adults (n = 20, 18, 19, and 19, respectively) produce names for 93 color swatches. The results showed that children of all ages were able to apply most of the chromatic words to the colors close to the center of each category, but even 5-year-olds struggle to delineate the boundaries between the words. Furthermore, the model analyses revealed that broad-covering and high-frequency words are mapped to the center of the lexical category earlier. However, cross-individual consistency in adults' use contributed most strongly for the adult-like boundary delineation. The results suggest that the process of system construction consists of at least two steps (i.e., mapping words to their category center and finding appropriate boundaries between neighboring words), with the quantity and the quality of the input contributing differently to the steps.