To better understand the prefrontal circuitry that putatively supports executive functions, such as those involved in switching tasks, we asked whether a current task set is open equally to receiving information from any sensory modality or if it is to some degree modality-specific. Subjects were presented with a sequence of digits to be classified as either odd/even or greater/less than five. The digits were either auditory or visual, with the modality varying randomly. Results demonstrated a reaction time (RT) cost associated with switching between tasks and also an unexpected cost of switching between modalities. When both modality and task switched, the two costs were greater than either of the costs alone, but significantly less than predicted simply by summing the two costs together (i.e., they were underadditive). These data indicate that the frontal mechanisms that allow for a switch in task are only partially modality-specific. Current theories of executive function must be adapted to account for this finding. We also suggest that the present paradigm is amenable to future research aimed at determining precisely how modalities are linked within a task set.