Speech perception involves fusion of multiple sensory inputs, but fusion is not automatic, likely depending on several external and internal factors (e.g., attention, noise, age). In this study, we exploited a specific paradigm in which a short audiovisual context made of coherent or incoherent speech material is displayed before an incongruent audiovisual target likely to provide fusion (McGurk & MacDonald, 1976). We confirmed that incoherent context leads to unbinding, that is, a reduction in the amount of fusion. Importantly, adding acoustic noise in the context though not in the target increases fusion. This suggests that listeners systematically evaluate the reliability of their sensory channels and weight them accordingly in the fusion process. We also showed that older participants display more unbinding than younger participants. We discuss the potential consequences concerning people's ability to understand speech in adverse conditions and relate our findings to a "Binding-and-Fusion" model of audiovisual speech perception.