The ability to switch between various representations is an invaluable problem-solving skill in physics. In addition, research has shown that using multiple representations can greatly enhance a person's understanding of mathematical and physical concepts. This paper describes a study of student difficulties regarding interpreting, constructing, and switching between representations of vector fields, using both qualitative and quantitative methods. We first identified to what extent students are fluent with the use of field vector plots, field line diagrams, and symbolic expressions of vector fields by conducting individual student interviews and analyzing in-class student activities. Based on those findings, we designed the Vector Field Representations test, a free response assessment tool that has been given to 196 second- and third-year physics, mathematics, and engineering students from four different universities. From the obtained results we gained a comprehensive overview of typical errors that students make when switching between vector field representations. In addition, the study allowed us to determine the relative prevalence of the observed difficulties. Although the results varied greatly between institutions, a general trend revealed that many students struggle with vector addition, fail to recognize the field line density as an indication of the magnitude of the field, confuse characteristics of field lines and equipotential lines, and do not choose the appropriate coordinate system when writing out mathematical expressions of vector fields.