Four experiments were conducted to investigate the interplay between context and tone of voice in the perception of sarcasm. These experiments emphasized the role of contrast effects in sarcasm perception exclusively by means of auditory stimuli whereas most past research has relied on written material. In all experiments, a positive or negative computer-generated context spoken in a flat emotional tone was followed by a literally positive statement spoken in a sincere or sarcastic tone of voice. Participants indicated for each statement whether the intonation was sincere or sarcastic. In Experiment 1, a congruent context/tone of voice pairing (negative/sarcastic, positive/sincere) produced fast response times and proportions of sarcastic responses in the direction predicted by the tone of voice. Incongruent pairings produced mid-range proportions and slower response times. Experiment 2 introduced ambiguous contexts to determine whether a lower context/statements contrast would affect the proportion of sarcastic responses and response time. Results showed the expected findings for proportions (values between those obtained for congruent and incongruent pairings in the direction predicted by the tone of voice). However, response time failed to produce the predicted pattern, suggesting potential issues with the choice of stimuli. Experiments 3 and 4 extended the results of Experiments 1 and 2, respectively, to auditory stimuli based on written vignettes used in neuropsychological assessment. Results were exactly as predicted by contrast effects in both experiments. Taken together, the findings suggest that both context and tone influence how sarcasm is perceived while supporting the importance of contrast effects in sarcasm perception.