How does the concurrent use of language affect perception and memory for exemplars? Labels cue more general category information than a specific exemplar. Applying labels can affect the resulting memory for an exemplar. Here 3 alternative hypotheses are proposed for the role of labeling an exemplar at encoding: (a) labels distort memory toward the label prototype, (b) labels guide the level of specificity needed in the current context, and (c) labels direct attention to the label's referent among all possible features within a visual scene. University students were shown hues on object silhouettes that they either labeled with basic color categories, made preference judgments about, or indicated the animacy of its category. Experiments 1 and 2 established that there are response shifts toward the category prototype regardless of labeling, showing a pervasive influence of category knowledge on response bias. They also established an effect of labeling whereby labeling decreases the magnitude of shifts. Experiments 3 and 4 investigated the uniqueness and necessity of language in causing the decreased shift--neither of which proved to be the case. Overall, category-relative bias was pervasive and labeling appears to direct attention to the feature resulting in less biased memory. The results highlight that the context at encoding affects how memory is formed.