Currently, an increasing number of educational institutions are redefining second/foreign language (L2) classrooms by enhancing--or even replacing--traditional face-to-face (FTF) instruction with computer-assisted language learning (CALL). However, are these curricular decisions supported by research? Overall, a cursory review of empirical studies appears to provide strong support for the use of CALL. Studies claim that technology can successfully promote learning in several areas of L2 acquisition, including grammar, and that it can even outperform FTF instruction in some cases. However, the methodological validity of this body of research has never been thoroughly assessed and, consequently, we ignore how much confidence we can place in its results. To address this, the present study analyzes the internal and external validity of 16 recent developmental studies. Based on this analysis, we conclude that at this point no strong argument can be made about whether or not the medium matters in L2 development, until further empirical studies with more robust research designs fully address the comparative effects of CALL vs. FTF instruction, including the role of some possibly intervening variables such as task type, time-on-task, or modality. To this end, we provide methodological recommendations that may raise the internal and external validity of future research designs as the empirical interest in CALL continues to blossom.