Purpose: Better auditory prostheses and earlier interventions have led to remarkable improvements in spoken language abilities for children with hearing loss (HL), but these children often still struggle academically. This study tested a hypothesis for why this may be, proposing that the language of school becomes increasingly disconnected from everyday discourse, requiring greater reliance on bottom-up phonological structure, and children with HL have difficulty recovering that structure from the speech signal. Participants: One hundred nineteen fourth graders participated--48 with normal hearing (NH), 19 with moderate losses who used hearing aids (HAs), and 52 with severe-to-profound losses who used cochlear implants (CIs). Method: Three analyses were conducted. (1) Sentences with malapropisms were created, and children's abilities to recognize them were assessed; (2) Factors contributing to those abilities were evaluated, including phonological awareness, phonological processing, vocabulary, verbal working memory, and oral narratives; and (3) Teachers' ratings of students' academic competence were obtained, and factors accounting for those ratings were evaluated, including the five listed above, along with word reading and reading comprehension. Results: (1) Children with HAs and CIs performed more poorly on malapropism recognition than children with NH, but similarly to each other; (2) All children with HL demonstrated large phonological deficits, but they were especially large for children with CIs. Phonological awareness explained the most variance in malapropism recognition for children with CIs. Vocabulary knowledge explained malapropism recognition for children with NH or HAs, but other factors also contributed; and (3) Teachers rated academic competence for children with CIs more poorly than for children with NH or HAs, and variance in those ratings for children with CIs were primarily explained by malapropism scores. Conclusion: Children with HL have difficulty recognizing acoustic--phonetic detail in the speech signal, and that constrains their abilities to follow conversations in academic settings, especially if HL is severe enough to require CIs.