The size congruity effect (SiCE) shows that number and physical size interact as magnitudes. That is, response times are faster when number and size are congruent (e.g., 2 4) than when they are incongruent (e.g., 2 4). A shared representational system has been the most influential account for the SiCE. Recently, this account has been challenged by findings showing that the SiCE may be influenced by attention. The attentional contribution to the SiCE suggests that the effect is produced by an attention capture effect to the larger stimulus. Even though plausible, the attentional account overlooks 2 important factors in the study of magnitudes, namely, task (numerical vs. physical) and polarity of instructions (choose the larger vs. the smaller). We studied the influence of these factors using a size congruity task. Experiment 1 showed that the SiCE was modulated by task and instructions. In Experiment 2, we used a new set of numbers to examine a possible influence of the so-called end effect (i.e., responses to the smallest and to the largest numbers may not require number comparison). Experiment 2 successfully replicated the pattern of Experiment 1. We suggest that both feature saliency and long-term semantic processes modulate the SiCE.