Background: Accurate identification of nonverbal emotional cues is essential to successful social interactions, yet most research is limited to emotional face expression labeling. Little research focuses on the processing of emotional prosody, or tone of verbal speech, in clinical populations. Methods: Using the Diagnostic Analysis of Nonverbal Accuracy, the current study examined whether youths with pediatric-onset bipolar disorder (BD) and/or those with chronic and severe irritability (i.e. the severe mood dysregulation phenotype) are impaired in their ability to identify the emotional prosody of a spoken sentence with neutral content. Results: Youths with severe mood dysregulation (n = 67) performed more poorly than healthy comparison children (n = 57), even when the sample was limited to unmedicated patients. Medicated BD youths (n = 52) exhibited impairment relative to healthy comparison children. No interactions between group and emotion were observed, suggesting that emotional prosody labeling problems may represent a general deficit in chronically irritable youths and in medicated youths with BD. Conclusion: In concert with previously documented facial emotion labeling deficits, difficulties ascertaining the correct emotional tone of a spoken sentence may contribute to emotion dysregulation in chronically irritable children, and possibly also in youths with BD.