We investigated the effect of modeling and visualization resources on upper-division, undergraduate and graduate students' performance on an open-ended assessment of their understanding of physical hydrology. The students were enrolled in one of five sections of a physical hydrology course. In two of the sections, students completed homework problems and projects using only Excel (Microsoft, Redmond, WA) or MATLAB (MathWorks, Natick, MA) as modeling resources, and in the other three sections, some of the homework exercises were replaced with modeling and visualization activities using the interactive modeling software COMSOL Multiphysics (COMSOL, Burlington, MA). Other aspects of the course (instructor, syllabus coverage, lectures, and textbook) remained the same throughout the study. We performed a repeated-measures analysis of variance, which showed that gains from pretest to posttest were statistically significant overall and were independent of the section in which students were enrolled for all but one component on the assessment. In that case, students who did not have access to the COMSOL modules marginally outperformed the others, but not to the required level of statistical significance. These results were complemented by a qualitative investigation of students' interaction with the modeling software. We interviewed a subset of students and assigned codes to themes that arose when we analyzed the resulting transcripts. This process allowed us to develop a theory of how students were interacting with the modeling and visualization resources. A significant theme was the issue of ''scaffolding,'' or supports, with both positive and negative consequences for students, depending on their personal preferences and previous experience.