Women of color are deeply underrepresented in physics. Between 2002 and 2012, only 1% of graduating physics majors were Black women and another 1% were Latinas; only 61 American Indian women total completed degrees in physics in those years (out of 48,000 physics majors). This isolation can lead to additional obstacles that women of color majoring in physics must face above and beyond the challenging material. In this article we draw on qualitative findings to describe common obstacles women of color face. However, departments can take deliberate steps so that underrepresentation need not turn into loneliness and isolation. We describe the characteristics of a department where women of color report that they are thriving. We end with concrete steps physics faculty can take to support women physics majors of color.