Literature on auditory distraction has generally focused on the effects of particular kinds of sounds on attention to target stimuli. In support of extensive previous findings that have demonstrated the special role of language as an auditory distractor, we found that a concurrent speech stream impaired recall of a short lecture, especially for verbatim language. But impaired recall effects were also found with a variety of nonlinguistic noises, suggesting that neither type of noise nor amplitude and duration of noise are adequate predictors of distraction. Rather, distraction occurred when it was difficult for a listener to process sounds and assemble coherent, differentiable streams of input, one task-salient and attended and the other task-irrelevant and inhibited. In 3 experiments, the effects of auditory distractors during a short spoken lecture were tested. Participants recalled details of the lecture and also reported their opinions of the sound quality. Our findings suggest that distractors that are difficult to designate as either task related or environment related (and therefore irrelevant) draw cognitive processing resources away from a target speech stream during a listening task, impairing recall.