One prominent feature of adaptive cognition in humans is the ability to flexibly adjust to changing task demands. In this respect, context-specific proportion congruency (CSPC) effects describe the phenomenon that participants learn to adapt to contexts of frequently occurring conflicts even when the upcoming context cannot be anticipated. Here, we aim to provide evidence that such CSPC effects strongly depend on the affective valence of the context. Participants had to categorize pictures as animals versus humans (Experiment 1), words as male versus female (Experiment 2), and food pictures as depicting cold or hot dishes/beverages (Experiment 3). Stimuli were laterally presented to the left or right, creating Simon-like response interference. Half of the stimuli were of positive valence, half of negative valence with valence always being task-irrelevant. Positive or negative stimuli were associated with a high proportion of incongruent trials and a low proportion of congruent trials, respectively. In all three experiments, a CSPC effect was only significant when the high proportion of incongruent trials was associated with positive stimuli but reduced or even absent when the high proportion of incongruent trials was associated with negative stimuli. The findings further speak to the idea that it is the aversive nature of conflict stimuli that functions as a trigger for control adaptation. And such an unexpected aversive conflict signal looms larger when presented in a positive context than in a negative context.