In 2 variants of the color-word Stroop task, we compared 5 types of color-neutral distractors--real words (e.g., "HAT"), pseudowords (e.g., "HIX"), consonant strings (e.g., "HDK"), symbol strings (e.g., #$%), and a row of Xs (e.g., "XXX")--as well as incongruent color words (e.g., "GREEN" displayed in red). When participants named the color, relative to a row of Xs, words and pseudowords interfered equally and more than the consonant strings, which in turn interfered more than the symbols. In contrast, when participants identified the color by manual key-press responses, all 5 types of neutral strings produced equal color response latencies. In both tasks, the incongruent color words produced robust interference relative to the color-neutral words. Reaction time (RT) distribution analyses showed that all interference effects (relative to the row of Xs) increased across the quantiles. We interpret these results in terms of an evidence accumulation process in which the interfering distractor reduces the effective rate of evidence accumulation for the color target. We take the results to argue that the task of reading, even when triggered unintentionally, is not an invariant process driven solely by the stimulus properties, and is instead guided by the task goal.