This study assessed the role different kinds of secondary tasks play for researching the modality effect of cognitive load theory. Ninety-six university students worked with a computer-based training program for approximately 13 min and had to fulfill an additional secondary task. In a 2 x 2 factorial design, modality of information presentation (within factor) and design of secondary task (between factor) were varied. Students of both experimental groups learned with visual-only and audiovisual information presentation. The secondary task consisted of monitoring an object either displayed spatially contiguous (monitoring the screen background color, N = 46) or spatially non-contiguous (monitoring a letter color in the upper part of the screen, N = 50). Reaction times on this secondary task were used to measure cognitive load. Results show that the modality effect only appears with the spatially non-contiguous task but not with the spatially contiguous task. We interpret this effect as due to only partial utilization of working memory capacity by the combination of primary task and spatially contiguous secondary task. The results highlight the importance of an appropriate secondary task design when investigating the modality effect but also not to overgeneralize multimedia design guidelines.