Purpose: We aim to identify the possible sources for age-related differences in the perception of emotion in speech, focusing on the distinct roles of semantics (words) and prosody (tone of speech) and their interaction. Method: We implement the Test for Rating of Emotions in Speech (Ben-David, Multani, Shakuf, Rudzicz, & van Lieshout, 2016). Forty older and 40 younger adults were presented with spoken sentences made of different combinations of 5 emotional categories (anger, fear, happiness, sadness, and neutral) presented in the prosody and semantics. In separate tasks, listeners were asked to attend to the sentence as a whole, integrating both speech channels, or to focus on 1 channel only (prosody/semantics). Their task was to rate how much they agree the sentence is conveying a predefined emotion. Results: (a) Identification of emotions: both age groups identified presented emotions. (b) Failure of selective attention: both age groups were unable to selectively attend to 1 channel when instructed, with slightly larger failures for older adults. (c) Integration of channels: younger adults showed a bias toward prosody, whereas older adults showed a slight bias toward semantics. Conclusions: Three possible sources are suggested for age-related differences: (a) underestimation of the emotional content of speech, (b) slightly larger failures to selectively attend to 1 channel, and (c) different weights assigned to the 2 speech channels.