In this essay, I draw on the socio-historical context of free speech and hate speech in U.S. public higher education and the concepts of color-evasiveness and free speech ideology to conceptualize a "color-evasive free speech ideology." The ideology I conceptualize is characterized by a prevailing belief that protecting and defending free speech rights is the only way to ensure democracy and equality, regardless of the racial harm and violence enacted by speech, and to the degree that those who challenge racist hate speech should be punished as a threat to free speech. I then explore three recent events to contextualize color-evasive free speech ideology: higher education professional organizations' responses to the release of Executive Order 13864 (Improving Free Inquiry, Transparency, and Accountability at Colleges and Universities) in March 2019 and two institutional-level examples of controlling student behavior through selective and racialized protection of free speech. I discuss the examples to illustrate how color-evasive free speech ideology upholds white supremacy and conclude with implications for scholars and practitioners, urging a critical troubling of color-evasive free speech ideology in future research and practice.